[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:51:54 dist-test-slave-8p8c krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:51:54 dist-test-slave-8p8c krb5kdc[395](info): set up 2 sockets
Jun 20 02:51:54 dist-test-slave-8p8c krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 02:51:55.592366 388 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.011s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:51:55 dist-test-slave-8p8c krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:51:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:51:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:51:55.783840 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44097
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:51:56.122768 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:51:56.123062 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:51:56.123207 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:51:56.131023 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:51:56.131181 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:51:56.131258 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:51:56.131320 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:51:56.131357 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:51:56.141510 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:51:56.142851 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:51:56.144357 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:51:56.158335 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:51:56.158514 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:51:56.158412 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:51:56.158476 408 server_base.cc:1034] running on GCE node
I20260620 02:51:56.159691 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:51:56.161259 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:51:56.162736 408 hybrid_clock.cc:648] HybridClock initialized: now 1781923916162653 us; error 80 us; skew 500 ppm
I20260620 02:51:56.163122 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:51:56 dist-test-slave-8p8c krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:51:56.171458 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:51:56.173758 408 webserver.cc:460] Webserver started at http://127.0.97.62:40173/ using document root <none> and password file <none>
I20260620 02:51:56.174593 408 fs_manager.cc:359] Metadata directory not provided
I20260620 02:51:56.174723 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:51:56.175055 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:51:56.178310 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "d3b242a1a9b74d5d80b72c7b82241f1b"
format_stamp: "Formatted at 2026-06-20 02:51:56 on dist-test-slave-8p8c"
server_key: "e4f530e7fe8e7e1538be20f95749f027"
server_key_iv: "5631a32ec19f723a81ce11e4fe54c180"
server_key_version: "encryptionkey@0"
I20260620 02:51:56.179352 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "d3b242a1a9b74d5d80b72c7b82241f1b"
format_stamp: "Formatted at 2026-06-20 02:51:56 on dist-test-slave-8p8c"
server_key: "e4f530e7fe8e7e1538be20f95749f027"
server_key_iv: "5631a32ec19f723a81ce11e4fe54c180"
server_key_version: "encryptionkey@0"
I20260620 02:51:56.185963 408 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 02:51:56.189815 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:51:56.191437 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 02:51:56.191615 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "d3b242a1a9b74d5d80b72c7b82241f1b"
format_stamp: "Formatted at 2026-06-20 02:51:56 on dist-test-slave-8p8c"
server_key: "e4f530e7fe8e7e1538be20f95749f027"
server_key_iv: "5631a32ec19f723a81ce11e4fe54c180"
server_key_version: "encryptionkey@0"
I20260620 02:51:56.191807 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:51:56.213435 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:51:56.218670 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:51:56.219033 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:51:56.240375 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44097
I20260620 02:51:56.240410 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44097 every 8 connection(s)
I20260620 02:51:56.241843 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:51:56.246052 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:51:56.252816 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 408
I20260620 02:51:56.253154 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:51:56.255170 388 external_mini_cluster.cc:1428] Setting key cedf1acdd4a4543f12940ad37d63da0d
I20260620 02:51:56.256100 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3b242a1a9b74d5d80b72c7b82241f1b: Bootstrap starting.
I20260620 02:51:56.259882 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3b242a1a9b74d5d80b72c7b82241f1b: Neither blocks nor log segments found. Creating new log.
I20260620 02:51:56.261620 475 log.cc:824] T 00000000000000000000000000000000 P d3b242a1a9b74d5d80b72c7b82241f1b: Log is configured to *not* fsync() on all Append() calls
I20260620 02:51:56.264372 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3b242a1a9b74d5d80b72c7b82241f1b: No bootstrap required, opened a new log
I20260620 02:51:56.272967 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3b242a1a9b74d5d80b72c7b82241f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3b242a1a9b74d5d80b72c7b82241f1b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44097 } }
I20260620 02:51:56.273380 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3b242a1a9b74d5d80b72c7b82241f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:51:56.273509 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3b242a1a9b74d5d80b72c7b82241f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3b242a1a9b74d5d80b72c7b82241f1b, State: Initialized, Role: FOLLOWER
I20260620 02:51:56.274235 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3b242a1a9b74d5d80b72c7b82241f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3b242a1a9b74d5d80b72c7b82241f1b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44097 } }
I20260620 02:51:56.274524 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3b242a1a9b74d5d80b72c7b82241f1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:51:56.274679 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3b242a1a9b74d5d80b72c7b82241f1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:51:56.274866 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3b242a1a9b74d5d80b72c7b82241f1b [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:51:56.276983 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3b242a1a9b74d5d80b72c7b82241f1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3b242a1a9b74d5d80b72c7b82241f1b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44097 } }
I20260620 02:51:56.277503 475 leader_election.cc:302] T 00000000000000000000000000000000 P d3b242a1a9b74d5d80b72c7b82241f1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3b242a1a9b74d5d80b72c7b82241f1b; no voters:
I20260620 02:51:56.278232 475 leader_election.cc:288] T 00000000000000000000000000000000 P d3b242a1a9b74d5d80b72c7b82241f1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:51:56.278434 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3b242a1a9b74d5d80b72c7b82241f1b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:51:56.280015 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3b242a1a9b74d5d80b72c7b82241f1b [term 1 LEADER]: Becoming Leader. State: Replica: d3b242a1a9b74d5d80b72c7b82241f1b, State: Running, Role: LEADER
I20260620 02:51:56.280694 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3b242a1a9b74d5d80b72c7b82241f1b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:51:56.280570 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3b242a1a9b74d5d80b72c7b82241f1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3b242a1a9b74d5d80b72c7b82241f1b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44097 } }
I20260620 02:51:56.284056 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3b242a1a9b74d5d80b72c7b82241f1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3b242a1a9b74d5d80b72c7b82241f1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3b242a1a9b74d5d80b72c7b82241f1b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44097 } } }
I20260620 02:51:56.284485 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3b242a1a9b74d5d80b72c7b82241f1b [sys.catalog]: This master's current role is: LEADER
I20260620 02:51:56.287900 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3b242a1a9b74d5d80b72c7b82241f1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3b242a1a9b74d5d80b72c7b82241f1b. Latest consensus state: current_term: 1 leader_uuid: "d3b242a1a9b74d5d80b72c7b82241f1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3b242a1a9b74d5d80b72c7b82241f1b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44097 } } }
I20260620 02:51:56.288277 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3b242a1a9b74d5d80b72c7b82241f1b [sys.catalog]: This master's current role is: LEADER
Jun 20 02:51:56 dist-test-slave-8p8c krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:51:56.298182 488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:51:56.306891 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:51:56.259046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45166 (local address 127.0.97.62:44097)
0620 02:51:56.259854 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0620 02:51:56.259875 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 02:51:56.261045 (+ 1170us) server_negotiation.cc:407] Connection header received
0620 02:51:56.272442 (+ 11397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:51:56.272470 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:51:56.272934 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:51:56.273472 (+ 538us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:51:56.276833 (+ 3361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:51:56.278734 (+ 1901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:51:56.281393 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:51:56.282340 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:51:56.289076 (+ 6736us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:51:56.291701 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:51:56.291717 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:51:56.291746 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:51:56.291805 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:51:56.297942 (+ 6137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:51:56.299784 (+ 1842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:51:56.299800 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:51:56.299818 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:51:56.300133 (+ 315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:51:56.300957 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:51:56.300963 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:51:56.300966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:51:56.301812 (+ 846us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:51:56.304207 (+ 2395us) server_negotiation.cc:1035] Waiting for connection context
0620 02:51:56.304526 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 02:51:56.305012 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":376,"threads_started":1}
I20260620 02:51:56.307224 488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:51:56.318192 488 catalog_manager.cc:1347] Generated new cluster ID: 8012a5242e4448eeb5e8bfb2d4da210c
I20260620 02:51:56.318320 488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:51:56.349315 488 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:51:56.351076 488 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:51:56.361177 488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3b242a1a9b74d5d80b72c7b82241f1b: Generated new TSK 0
I20260620 02:51:56.362310 488 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:51:56.414729 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44097
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:51:56.766283 502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:51:56.766628 502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:51:56.766773 502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:51:56.775000 502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:51:56.775224 502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:51:56.775418 502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:51:56.785456 502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44097
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:51:56.786813 502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:51:56.788223 502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:51:56.802567 510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:51:56.802546 508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:51:56.802644 507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:51:56.803702 502 server_base.cc:1034] running on GCE node
I20260620 02:51:56.804427 502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:51:56.805579 502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:51:56.806804 502 hybrid_clock.cc:648] HybridClock initialized: now 1781923916806749 us; error 49 us; skew 500 ppm
I20260620 02:51:56.807183 502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:51:56 dist-test-slave-8p8c krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:51:56.810520 502 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:51:56.812203 502 webserver.cc:460] Webserver started at http://127.0.97.1:41759/ using document root <none> and password file <none>
I20260620 02:51:56.812916 502 fs_manager.cc:359] Metadata directory not provided
I20260620 02:51:56.813014 502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:51:56.813367 502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:51:56.816337 502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-0/data/instance:
uuid: "6fc52c43397d4ac790f108685cf5987b"
format_stamp: "Formatted at 2026-06-20 02:51:56 on dist-test-slave-8p8c"
server_key: "0a44aaa5592ef52d9959daefdc83de5a"
server_key_iv: "ed5114f18ab142de1dd8c9359eec74f8"
server_key_version: "encryptionkey@0"
I20260620 02:51:56.817111 502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6fc52c43397d4ac790f108685cf5987b"
format_stamp: "Formatted at 2026-06-20 02:51:56 on dist-test-slave-8p8c"
server_key: "0a44aaa5592ef52d9959daefdc83de5a"
server_key_iv: "ed5114f18ab142de1dd8c9359eec74f8"
server_key_version: "encryptionkey@0"
I20260620 02:51:56.822220 502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 02:51:56.826221 517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:51:56.827732 502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:51:56.827965 502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-0/wal
uuid: "6fc52c43397d4ac790f108685cf5987b"
format_stamp: "Formatted at 2026-06-20 02:51:56 on dist-test-slave-8p8c"
server_key: "0a44aaa5592ef52d9959daefdc83de5a"
server_key_iv: "ed5114f18ab142de1dd8c9359eec74f8"
server_key_version: "encryptionkey@0"
I20260620 02:51:56.828150 502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:51:56.843447 502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:51:56.847988 502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:51:56.848325 502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:51:56.849354 502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:51:56.850829 502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:51:56.850908 502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:51:56.851038 502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:51:56.851111 502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:51:56.890798 502 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37289
I20260620 02:51:56.890853 629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37289 every 8 connection(s)
I20260620 02:51:56.892168 502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:51:56.902822 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 502
I20260620 02:51:56.903088 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:51:56.903770 388 external_mini_cluster.cc:1428] Setting key 206e808f7304df07b373f0c5f6a9f470
Jun 20 02:51:56 dist-test-slave-8p8c krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:51:56.915577 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:51:56.895391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47653 (local address 127.0.97.62:44097)
0620 02:51:56.895922 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0620 02:51:56.895928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:51:56.897113 (+ 1185us) server_negotiation.cc:407] Connection header received
0620 02:51:56.899120 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:51:56.899125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:51:56.899214 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:51:56.899335 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:51:56.900849 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:51:56.901674 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:51:56.903496 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:51:56.903939 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:51:56.904103 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:51:56.908082 (+ 3979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:51:56.908088 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:51:56.908091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:51:56.908124 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:51:56.910978 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:51:56.912357 (+ 1379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:51:56.912365 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:51:56.912371 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:51:56.912528 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:51:56.913069 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:51:56.913076 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:51:56.913083 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:51:56.913488 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:51:56.913659 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 02:51:56.914807 (+ 1148us) server_negotiation.cc:299] Negotiation successful
0620 02:51:56.915106 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":240,"threads_started":1}
I20260620 02:51:56.916610 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:51:56.896080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44097 (local address 127.0.97.1:47653)
0620 02:51:56.896798 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0620 02:51:56.896843 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 02:51:56.898845 (+ 2002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:51:56.899581 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:51:56.899590 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:51:56.900099 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:51:56.900662 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:51:56.900678 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:51:56.901856 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:51:56.901869 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:51:56.903343 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:51:56.903355 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:51:56.904191 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:51:56.904200 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:51:56.904448 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:51:56.905304 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:51:56.905328 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:51:56.907748 (+ 2420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:51:56.911266 (+ 3518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:51:56.911277 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:51:56.911293 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:51:56.912167 (+ 874us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:51:56.912703 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:51:56.912708 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:51:56.912711 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:51:56.912929 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:51:56.913879 (+ 950us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:51:56.913887 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:51:56.914292 (+ 405us) client_negotiation.cc:769] Sending connection context
0620 02:51:56.914657 (+ 365us) client_negotiation.cc:240] Negotiation successful
0620 02:51:56.915342 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":298,"threads_started":1}
I20260620 02:51:56.919306 630 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44097
I20260620 02:51:56.919674 630 heartbeater.cc:464] Registering TS with master...
I20260620 02:51:56.920687 630 heartbeater.cc:511] Master 127.0.97.62:44097 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:51:56.923485 440 ts_manager.cc:194] Registered new tserver with Master: 6fc52c43397d4ac790f108685cf5987b (127.0.97.1:37289)
I20260620 02:51:56.925195 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:51:56.957474 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44097
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:51:57.299724 636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:51:57.300005 636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:51:57.300170 636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:51:57.307889 636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:51:57.308020 636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:51:57.308192 636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:51:57.317940 636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44097
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:51:57.319187 636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:51:57.320539 636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:51:57.333894 641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:51:57.333917 642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:51:57.333938 644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:51:57.335150 636 server_base.cc:1034] running on GCE node
I20260620 02:51:57.335750 636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:51:57.336724 636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:51:57.337945 636 hybrid_clock.cc:648] HybridClock initialized: now 1781923917337894 us; error 50 us; skew 500 ppm
I20260620 02:51:57.338270 636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:51:57 dist-test-slave-8p8c krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:51:57.341359 636 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:51:57.343261 636 webserver.cc:460] Webserver started at http://127.0.97.2:41813/ using document root <none> and password file <none>
I20260620 02:51:57.344025 636 fs_manager.cc:359] Metadata directory not provided
I20260620 02:51:57.344122 636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:51:57.344369 636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:51:57.347442 636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-1/data/instance:
uuid: "808a7d45f5204a168756b06d6572cd4c"
format_stamp: "Formatted at 2026-06-20 02:51:57 on dist-test-slave-8p8c"
server_key: "c099459aad77be238f44e5127ef0c8f6"
server_key_iv: "600f7ab1d3f8a5b4a0175d7ef9972feb"
server_key_version: "encryptionkey@0"
I20260620 02:51:57.348273 636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-1/wal/instance:
uuid: "808a7d45f5204a168756b06d6572cd4c"
format_stamp: "Formatted at 2026-06-20 02:51:57 on dist-test-slave-8p8c"
server_key: "c099459aad77be238f44e5127ef0c8f6"
server_key_iv: "600f7ab1d3f8a5b4a0175d7ef9972feb"
server_key_version: "encryptionkey@0"
I20260620 02:51:57.353737 636 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 02:51:57.357432 651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:51:57.358933 636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 02:51:57.359103 636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-1/wal
uuid: "808a7d45f5204a168756b06d6572cd4c"
format_stamp: "Formatted at 2026-06-20 02:51:57 on dist-test-slave-8p8c"
server_key: "c099459aad77be238f44e5127ef0c8f6"
server_key_iv: "600f7ab1d3f8a5b4a0175d7ef9972feb"
server_key_version: "encryptionkey@0"
I20260620 02:51:57.359306 636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:51:57.372962 636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:51:57.377604 636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:51:57.377890 636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:51:57.378878 636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:51:57.380142 636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:51:57.380215 636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:51:57.380358 636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:51:57.380420 636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:51:57.417356 636 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46881
I20260620 02:51:57.417450 763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46881 every 8 connection(s)
I20260620 02:51:57.418823 636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:51:57.425606 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 636
I20260620 02:51:57.425802 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:51:57.426223 388 external_mini_cluster.cc:1428] Setting key eab36fb0875d9409a56ecf3854dae2dc
Jun 20 02:51:57 dist-test-slave-8p8c krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:51:57.440940 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:51:57.421432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41605 (local address 127.0.97.62:44097)
0620 02:51:57.421903 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0620 02:51:57.421909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:51:57.423357 (+ 1448us) server_negotiation.cc:407] Connection header received
0620 02:51:57.425060 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:51:57.425064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:51:57.425151 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:51:57.425274 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:51:57.427054 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:51:57.427902 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:51:57.429781 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:51:57.430666 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:51:57.430814 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:51:57.435062 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:51:57.435067 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:51:57.435070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:51:57.435099 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:51:57.437670 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:51:57.438647 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:51:57.438651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:51:57.438653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:51:57.438771 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:51:57.439263 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:51:57.439265 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:51:57.439267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:51:57.439510 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:51:57.439614 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 02:51:57.440532 (+ 918us) server_negotiation.cc:299] Negotiation successful
0620 02:51:57.440682 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":234,"threads_started":1}
I20260620 02:51:57.442538 767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:51:57.422015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44097 (local address 127.0.97.2:41605)
0620 02:51:57.423207 (+ 1192us) negotiation.cc:107] Waiting for socket to connect
0620 02:51:57.423253 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 02:51:57.424826 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:51:57.425460 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:51:57.425468 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:51:57.426156 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:51:57.426844 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:51:57.426870 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:51:57.428046 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:51:57.428051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:51:57.429368 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:51:57.429380 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:51:57.430863 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:51:57.430868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:51:57.431133 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:51:57.431823 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:51:57.431845 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:51:57.434805 (+ 2960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:51:57.437817 (+ 3012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:51:57.437829 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:51:57.437849 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:51:57.438522 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:51:57.438884 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:51:57.438890 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:51:57.438895 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:51:57.439145 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:51:57.439639 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:51:57.439650 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:51:57.440131 (+ 481us) client_negotiation.cc:769] Sending connection context
0620 02:51:57.440453 (+ 322us) client_negotiation.cc:240] Negotiation successful
0620 02:51:57.441022 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":334,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:51:57.444051 764 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44097
I20260620 02:51:57.444299 764 heartbeater.cc:464] Registering TS with master...
I20260620 02:51:57.444943 764 heartbeater.cc:511] Master 127.0.97.62:44097 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 02:51:57.446919 440 ts_manager.cc:194] Registered new tserver with Master: 808a7d45f5204a168756b06d6572cd4c (127.0.97.2:46881)
I20260620 02:51:57.448189 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:51:57.466106 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44097
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:51:57.794289 770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:51:57.794639 770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:51:57.795161 770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:51:57.802623 770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:51:57.802723 770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:51:57.802834 770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:51:57.812616 770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44097
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:51:57.813844 770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:51:57.815292 770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:51:57.829152 775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:51:57.829097 778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:51:57.829123 776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:51:57.830140 770 server_base.cc:1034] running on GCE node
I20260620 02:51:57.830801 770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:51:57.831816 770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:51:57.833036 770 hybrid_clock.cc:648] HybridClock initialized: now 1781923917832997 us; error 33 us; skew 500 ppm
I20260620 02:51:57.833381 770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:51:57 dist-test-slave-8p8c krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:51:57.837052 770 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:51:57.838560 770 webserver.cc:460] Webserver started at http://127.0.97.3:39657/ using document root <none> and password file <none>
I20260620 02:51:57.839200 770 fs_manager.cc:359] Metadata directory not provided
I20260620 02:51:57.839289 770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:51:57.839531 770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:51:57.842283 770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-2/data/instance:
uuid: "a8da3a1c0a544060a7b780365356e183"
format_stamp: "Formatted at 2026-06-20 02:51:57 on dist-test-slave-8p8c"
server_key: "8ea75b1f30807f535b993b683afd2169"
server_key_iv: "0c192f62a44090a560e39ad6f43e6fd0"
server_key_version: "encryptionkey@0"
I20260620 02:51:57.843066 770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a8da3a1c0a544060a7b780365356e183"
format_stamp: "Formatted at 2026-06-20 02:51:57 on dist-test-slave-8p8c"
server_key: "8ea75b1f30807f535b993b683afd2169"
server_key_iv: "0c192f62a44090a560e39ad6f43e6fd0"
server_key_version: "encryptionkey@0"
I20260620 02:51:57.848088 770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 02:51:57.851596 785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:51:57.853121 770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:51:57.853307 770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-2/wal
uuid: "a8da3a1c0a544060a7b780365356e183"
format_stamp: "Formatted at 2026-06-20 02:51:57 on dist-test-slave-8p8c"
server_key: "8ea75b1f30807f535b993b683afd2169"
server_key_iv: "0c192f62a44090a560e39ad6f43e6fd0"
server_key_version: "encryptionkey@0"
I20260620 02:51:57.853510 770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:51:57.882217 770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:51:57.887001 770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:51:57.887311 770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:51:57.888403 770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:51:57.889873 770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:51:57.889968 770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:51:57.890075 770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:51:57.890157 770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:51:57.927845 770 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39751
I20260620 02:51:57.927945 897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39751 every 8 connection(s)
I20260620 02:51:57.928998 630 heartbeater.cc:503] Master 127.0.97.62:44097 was elected leader, sending a full tablet report...
I20260620 02:51:57.929284 770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:51:57.932842 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 770
I20260620 02:51:57.933050 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781923914418765-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:51:57.933547 388 external_mini_cluster.cc:1428] Setting key a48d71351aaa557971b3114210d70b43
Jun 20 02:51:57 dist-test-slave-8p8c krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:51:57.953452 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:51:57.932549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37931 (local address 127.0.97.62:44097)
0620 02:51:57.932722 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 02:51:57.932726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:51:57.934444 (+ 1718us) server_negotiation.cc:407] Connection header received
0620 02:51:57.936673 (+ 2229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:51:57.936679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:51:57.936766 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:51:57.936891 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:51:57.938681 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:51:57.939482 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:51:57.941202 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:51:57.941595 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:51:57.941695 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:51:57.945908 (+ 4213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:51:57.945912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:51:57.945916 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:51:57.945948 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:51:57.948909 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:51:57.950015 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:51:57.950021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:51:57.950024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:51:57.950183 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:51:57.950855 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:51:57.950878 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:51:57.950880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:51:57.951212 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:51:57.951340 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 02:51:57.952652 (+ 1312us) server_negotiation.cc:299] Negotiation successful
0620 02:51:57.953029 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 02:51:57.954526 900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:51:57.933316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44097 (local address 127.0.97.3:37931)
0620 02:51:57.934239 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0620 02:51:57.934290 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 02:51:57.936390 (+ 2100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:51:57.937334 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:51:57.937342 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:51:57.937858 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:51:57.938504 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:51:57.938523 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:51:57.939702 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:51:57.939707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:51:57.940979 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:51:57.940990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:51:57.941799 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:51:57.941804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:51:57.942010 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:51:57.943089 (+ 1079us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:51:57.943113 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:51:57.945624 (+ 2511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:51:57.949125 (+ 3501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:51:57.949138 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:51:57.949158 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:51:57.949788 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:51:57.950341 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:51:57.950350 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:51:57.950354 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:51:57.950664 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:51:57.951365 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:51:57.951372 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:51:57.952178 (+ 806us) client_negotiation.cc:769] Sending connection context
0620 02:51:57.952549 (+ 371us) client_negotiation.cc:240] Negotiation successful
0620 02:51:57.953205 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":464,"threads_started":1}
I20260620 02:51:57.956395 898 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44097
I20260620 02:51:57.957150 898 heartbeater.cc:464] Registering TS with master...
I20260620 02:51:57.957973 898 heartbeater.cc:511] Master 127.0.97.62:44097 requested a full tablet report, sending...
I20260620 02:51:57.959975 440 ts_manager.cc:194] Registered new tserver with Master: a8da3a1c0a544060a7b780365356e183 (127.0.97.3:39751)
I20260620 02:51:57.960866 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37931
I20260620 02:51:57.963433 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 02:51:57 dist-test-slave-8p8c krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923917, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 02:51:58 dist-test-slave-8p8c krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923917, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:51:58.011631 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:51:57.994804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54102 (local address 127.0.97.1:37289)
0620 02:51:57.995639 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0620 02:51:57.995657 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 02:51:57.995689 (+ 32us) server_negotiation.cc:407] Connection header received
0620 02:51:57.995795 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:51:57.995802 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:51:57.996031 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:51:57.996171 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:51:57.997742 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:51:57.998983 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:51:58.000262 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:51:58.000851 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:51:58.000998 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:51:58.005647 (+ 4649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:51:58.005658 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:51:58.005674 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:51:58.005713 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:51:58.008729 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:51:58.009432 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:51:58.009441 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:51:58.009444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:51:58.009581 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:51:58.009999 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:51:58.010003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:51:58.010005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:51:58.010478 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:51:58.010705 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 02:51:58.011032 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 02:51:58.011233 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":452,"threads_started":1}
W20260620 02:51:58.013203 545 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54102
Jun 20 02:51:58 dist-test-slave-8p8c krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:51:58 dist-test-slave-8p8c krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:51:58.059088 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:51:58.044088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54116 (local address 127.0.97.1:37289)
0620 02:51:58.044315 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 02:51:58.044322 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:51:58.044463 (+ 141us) server_negotiation.cc:407] Connection header received
0620 02:51:58.044688 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:51:58.044693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:51:58.044767 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:51:58.044877 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:51:58.046243 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:51:58.047024 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:51:58.048235 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:51:58.048662 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:51:58.048792 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:51:58.053277 (+ 4485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:51:58.053282 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:51:58.053286 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:51:58.053322 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:51:58.056228 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:51:58.057047 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:51:58.057053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:51:58.057057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:51:58.057201 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:51:58.057586 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:51:58.057591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:51:58.057595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:51:58.057918 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:51:58.058186 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0620 02:51:58.058528 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 02:51:58.058756 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 02:51:58.060674 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 502
I20260620 02:51:58.074307 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:51:58.061763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54132 (local address 127.0.97.1:37289)
0620 02:51:58.062019 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 02:51:58.062024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:51:58.062095 (+ 71us) server_negotiation.cc:407] Connection header received
0620 02:51:58.062272 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:51:58.062276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:51:58.062337 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:51:58.062487 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:51:58.063731 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:51:58.064418 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:51:58.065621 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:51:58.066079 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:51:58.066174 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:51:58.068538 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:51:58.068543 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:51:58.068546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:51:58.068574 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:51:58.071786 (+ 3212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:51:58.072475 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:51:58.072479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:51:58.072481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:51:58.072610 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:51:58.072918 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:51:58.072921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:51:58.072923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:51:58.073164 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:51:58.073324 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 02:51:58.073727 (+ 403us) server_negotiation.cc:299] Negotiation successful
0620 02:51:58.073882 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 02:51:58.075093 625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:51:58.158691 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 502
I20260620 02:51:58.175220 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 636
Jun 20 02:51:58 dist-test-slave-8p8c krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:51:58.191540 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:51:58.176102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35816 (local address 127.0.97.2:46881)
0620 02:51:58.176610 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0620 02:51:58.176617 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:51:58.176641 (+ 24us) server_negotiation.cc:407] Connection header received
0620 02:51:58.176697 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:51:58.176704 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:51:58.176912 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:51:58.177096 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:51:58.178270 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:51:58.179421 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:51:58.180649 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:51:58.181275 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:51:58.181426 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:51:58.185528 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:51:58.185539 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:51:58.185554 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:51:58.185587 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:51:58.188554 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:51:58.189218 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:51:58.189225 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:51:58.189227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:51:58.189357 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:51:58.189874 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:51:58.189878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:51:58.189880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:51:58.190326 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:51:58.190697 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0620 02:51:58.190914 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 02:51:58.191139 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":255,"threads_started":1}
I20260620 02:51:58.192835 759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:51:58.275699 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 636
I20260620 02:51:58.299149 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 770
Jun 20 02:51:58 dist-test-slave-8p8c krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:51:58.316974 914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:51:58.300083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33548 (local address 127.0.97.3:39751)
0620 02:51:58.300728 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0620 02:51:58.300736 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:51:58.300764 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:51:58.300823 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:51:58.300829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:51:58.301052 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:51:58.301180 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:51:58.302623 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:51:58.303656 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:51:58.304999 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:51:58.305558 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:51:58.305659 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:51:58.310294 (+ 4635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:51:58.310306 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:51:58.310321 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:51:58.310359 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:51:58.313843 (+ 3484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:51:58.314779 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:51:58.314789 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:51:58.314791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:51:58.314930 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:51:58.315348 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:51:58.315351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:51:58.315354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:51:58.315738 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:51:58.316031 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 02:51:58.316357 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 02:51:58.316625 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":343,"threads_started":1}
I20260620 02:51:58.318457 893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:51:58.402684 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 770
I20260620 02:51:58.418699 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 408
I20260620 02:51:58.420051 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:51:58.526412 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 408
2026-06-20T02:51:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4067 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:51:58 dist-test-slave-8p8c krb5kdc[919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:51:58 dist-test-slave-8p8c krb5kdc[919](info): set up 2 sockets
Jun 20 02:51:58 dist-test-slave-8p8c krb5kdc[919](info): commencing operation
krb5kdc: starting...
W20260620 02:51:59.630514 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:51:59 dist-test-slave-8p8c krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:51:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:51:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:51:59.807266 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38129
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38129
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:00.119122 932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:00.119464 932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:00.119635 932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:00.127094 932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:52:00.127246 932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:00.127346 932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:00.127447 932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:52:00.127523 932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:52:00.137377 932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:00.138823 932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:00.140450 932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:00.153421 937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:00.153446 938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:00.153674 940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:00.154007 932 server_base.cc:1034] running on GCE node
I20260620 02:52:00.155083 932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:00.156565 932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:00.157827 932 hybrid_clock.cc:648] HybridClock initialized: now 1781923920157692 us; error 122 us; skew 500 ppm
I20260620 02:52:00.158219 932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:00 dist-test-slave-8p8c krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:00.162144 932 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:52:00.163872 932 webserver.cc:460] Webserver started at http://127.0.97.62:46607/ using document root <none> and password file <none>
I20260620 02:52:00.164626 932 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:00.164773 932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:00.165103 932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:00.168231 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "8699abb486544b6ab328141383762dc2"
format_stamp: "Formatted at 2026-06-20 02:52:00 on dist-test-slave-8p8c"
server_key: "3d29cd701d0e79f1a0fe8832d3f0407a"
server_key_iv: "6ae4ca2a1a2c6233ea2dbed4e9ec293e"
server_key_version: "encryptionkey@0"
I20260620 02:52:00.169028 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "8699abb486544b6ab328141383762dc2"
format_stamp: "Formatted at 2026-06-20 02:52:00 on dist-test-slave-8p8c"
server_key: "3d29cd701d0e79f1a0fe8832d3f0407a"
server_key_iv: "6ae4ca2a1a2c6233ea2dbed4e9ec293e"
server_key_version: "encryptionkey@0"
I20260620 02:52:00.174525 932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 02:52:00.178063 947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:00.179770 932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:52:00.179908 932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "8699abb486544b6ab328141383762dc2"
format_stamp: "Formatted at 2026-06-20 02:52:00 on dist-test-slave-8p8c"
server_key: "3d29cd701d0e79f1a0fe8832d3f0407a"
server_key_iv: "6ae4ca2a1a2c6233ea2dbed4e9ec293e"
server_key_version: "encryptionkey@0"
I20260620 02:52:00.180087 932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:00.201686 932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:00.206808 932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:00.207067 932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:00.225445 932 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38129
I20260620 02:52:00.225472 998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38129 every 8 connection(s)
I20260620 02:52:00.226998 932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:52:00.231158 999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:00.233752 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 932
I20260620 02:52:00.234117 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:52:00.235008 388 external_mini_cluster.cc:1428] Setting key 1703e75a372453db8ad4a218f9da6a50
I20260620 02:52:00.243884 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8699abb486544b6ab328141383762dc2: Bootstrap starting.
I20260620 02:52:00.248023 999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8699abb486544b6ab328141383762dc2: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:00.249397 999 log.cc:824] T 00000000000000000000000000000000 P 8699abb486544b6ab328141383762dc2: Log is configured to *not* fsync() on all Append() calls
Jun 20 02:52:00 dist-test-slave-8p8c krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:00.252684 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8699abb486544b6ab328141383762dc2: No bootstrap required, opened a new log
I20260620 02:52:00.259119 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:00.237036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52848 (local address 127.0.97.62:38129)
0620 02:52:00.237840 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:00.237852 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:00.237896 (+ 44us) server_negotiation.cc:407] Connection header received
0620 02:52:00.239630 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:00.239656 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:00.240111 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:00.240462 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:00.242283 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:00.243756 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:00.245547 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:00.246321 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:00.246546 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:00.251326 (+ 4780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:00.251335 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:00.251351 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:00.251391 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:00.254780 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:00.255530 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:00.255538 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:00.255548 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:00.255717 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:00.256116 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:00.256127 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:00.256129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:00.256670 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:00.257011 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:00.257525 (+ 514us) server_negotiation.cc:299] Negotiation successful
0620 02:52:00.257954 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":371,"threads_started":1}
I20260620 02:52:00.260742 999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8699abb486544b6ab328141383762dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8699abb486544b6ab328141383762dc2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38129 } }
I20260620 02:52:00.261199 999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8699abb486544b6ab328141383762dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:00.261304 999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8699abb486544b6ab328141383762dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8699abb486544b6ab328141383762dc2, State: Initialized, Role: FOLLOWER
I20260620 02:52:00.261934 999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8699abb486544b6ab328141383762dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8699abb486544b6ab328141383762dc2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38129 } }
I20260620 02:52:00.262198 999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8699abb486544b6ab328141383762dc2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:52:00.262355 999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8699abb486544b6ab328141383762dc2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:52:00.262568 999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8699abb486544b6ab328141383762dc2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:00.264657 999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8699abb486544b6ab328141383762dc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8699abb486544b6ab328141383762dc2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38129 } }
I20260620 02:52:00.265110 999 leader_election.cc:302] T 00000000000000000000000000000000 P 8699abb486544b6ab328141383762dc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8699abb486544b6ab328141383762dc2; no voters:
I20260620 02:52:00.265620 999 leader_election.cc:288] T 00000000000000000000000000000000 P 8699abb486544b6ab328141383762dc2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:52:00.265865 1004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8699abb486544b6ab328141383762dc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:00.266616 1004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8699abb486544b6ab328141383762dc2 [term 1 LEADER]: Becoming Leader. State: Replica: 8699abb486544b6ab328141383762dc2, State: Running, Role: LEADER
I20260620 02:52:00.267166 1004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8699abb486544b6ab328141383762dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8699abb486544b6ab328141383762dc2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38129 } }
I20260620 02:52:00.267797 999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8699abb486544b6ab328141383762dc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:52:00.270150 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8699abb486544b6ab328141383762dc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8699abb486544b6ab328141383762dc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8699abb486544b6ab328141383762dc2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38129 } } }
I20260620 02:52:00.270628 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8699abb486544b6ab328141383762dc2 [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:00.270459 1006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8699abb486544b6ab328141383762dc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8699abb486544b6ab328141383762dc2. Latest consensus state: current_term: 1 leader_uuid: "8699abb486544b6ab328141383762dc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8699abb486544b6ab328141383762dc2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38129 } } }
I20260620 02:52:00.270757 1006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8699abb486544b6ab328141383762dc2 [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:00.272171 1011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:52:00.278059 1011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:52:00.288368 1011 catalog_manager.cc:1347] Generated new cluster ID: 4430f8d0b4b34f3cb90f0fd629952689
I20260620 02:52:00.288547 1011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:52:00.322566 1011 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:52:00.323863 1011 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:52:00.330154 1011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8699abb486544b6ab328141383762dc2: Generated new TSK 0
I20260620 02:52:00.331089 1011 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:52:00.382244 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38129
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:00.716522 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:00.716814 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:00.716976 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:00.724284 1025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:00.724401 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:00.724632 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:52:00.734791 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38129
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:00.736110 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:00.737522 1025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:00.751554 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:00.751498 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:00.751526 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:00.753273 1025 server_base.cc:1034] running on GCE node
I20260620 02:52:00.753993 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:00.755239 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:00.756533 1025 hybrid_clock.cc:648] HybridClock initialized: now 1781923920756466 us; error 51 us; skew 500 ppm
I20260620 02:52:00.756929 1025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:00 dist-test-slave-8p8c krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:00.760977 1025 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:52:00.762671 1025 webserver.cc:460] Webserver started at http://127.0.97.1:36191/ using document root <none> and password file <none>
I20260620 02:52:00.763479 1025 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:00.763648 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:00.763981 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:00.767112 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-0/data/instance:
uuid: "9a6977eae9034f1a9e290689255669f7"
format_stamp: "Formatted at 2026-06-20 02:52:00 on dist-test-slave-8p8c"
server_key: "edc77f76a6408700f04333dc0d7eb7d3"
server_key_iv: "7444aa6c0570227288cad0573d472ff3"
server_key_version: "encryptionkey@0"
I20260620 02:52:00.768131 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-0/wal/instance:
uuid: "9a6977eae9034f1a9e290689255669f7"
format_stamp: "Formatted at 2026-06-20 02:52:00 on dist-test-slave-8p8c"
server_key: "edc77f76a6408700f04333dc0d7eb7d3"
server_key_iv: "7444aa6c0570227288cad0573d472ff3"
server_key_version: "encryptionkey@0"
I20260620 02:52:00.773867 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 02:52:00.777678 1040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:00.779165 1025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:52:00.779335 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-0/wal
uuid: "9a6977eae9034f1a9e290689255669f7"
format_stamp: "Formatted at 2026-06-20 02:52:00 on dist-test-slave-8p8c"
server_key: "edc77f76a6408700f04333dc0d7eb7d3"
server_key_iv: "7444aa6c0570227288cad0573d472ff3"
server_key_version: "encryptionkey@0"
I20260620 02:52:00.779536 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:00.807173 1025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:00.811820 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:00.812125 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:00.813145 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:52:00.814692 1025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:00.814806 1025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:00.814965 1025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:00.815039 1025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:00.850030 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44061
I20260620 02:52:00.850116 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44061 every 8 connection(s)
I20260620 02:52:00.851368 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:52:00.860814 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 1025
I20260620 02:52:00.861037 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:52:00.861634 388 external_mini_cluster.cc:1428] Setting key c7ed555c8c6aad2ada6919f627549df9
Jun 20 02:52:00 dist-test-slave-8p8c krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:00.873854 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:00.854079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43871 (local address 127.0.97.62:38129)
0620 02:52:00.854568 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:00.854573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:00.855648 (+ 1075us) server_negotiation.cc:407] Connection header received
0620 02:52:00.857487 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:00.857493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:00.857586 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:00.857706 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:00.859350 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:00.860061 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:00.861716 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:00.862330 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:00.862480 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:00.867093 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:00.867102 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:00.867111 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:00.867157 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:00.870152 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:00.871205 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:00.871210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:00.871213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:00.871363 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:00.871969 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:00.871973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:00.871975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:00.872258 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:00.872383 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:00.873409 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0620 02:52:00.873588 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":251,"threads_started":1}
I20260620 02:52:00.875499 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:00.854676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38129 (local address 127.0.97.1:43871)
0620 02:52:00.855458 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:00.855502 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:00.857218 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:00.857938 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:00.857947 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:00.858534 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:00.859180 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:00.859196 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:00.860205 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:00.860212 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:00.861561 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:00.861577 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:00.862604 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:00.862631 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:00.862917 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:00.863844 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:00.863877 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:00.866831 (+ 2954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:00.870296 (+ 3465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:00.870309 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:00.870332 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:00.871032 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:00.871488 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:00.871496 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:00.871503 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:00.871761 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:00.872388 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:00.872400 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:00.872931 (+ 531us) client_negotiation.cc:769] Sending connection context
0620 02:52:00.873279 (+ 348us) client_negotiation.cc:240] Negotiation successful
0620 02:52:00.873889 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":343,"threads_started":1}
I20260620 02:52:00.877128 1153 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38129
I20260620 02:52:00.877450 1153 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:00.878160 1153 heartbeater.cc:511] Master 127.0.97.62:38129 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:52:00.880452 964 ts_manager.cc:194] Registered new tserver with Master: 9a6977eae9034f1a9e290689255669f7 (127.0.97.1:44061)
I20260620 02:52:00.882082 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:52:00.906985 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38129
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:01.261508 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:01.261909 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:01.262080 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:01.271467 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:01.271637 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:01.271847 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:52:01.282267 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38129
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:01.283982 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:01.285435 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:01.299010 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:01.299036 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:01.299011 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:01.299832 1159 server_base.cc:1034] running on GCE node
I20260620 02:52:01.300381 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:01.301276 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:01.302480 1159 hybrid_clock.cc:648] HybridClock initialized: now 1781923921302424 us; error 38 us; skew 500 ppm
I20260620 02:52:01.303112 1159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:01 dist-test-slave-8p8c krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:01.306710 1159 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:52:01.308226 1159 webserver.cc:460] Webserver started at http://127.0.97.2:33239/ using document root <none> and password file <none>
I20260620 02:52:01.308929 1159 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:01.309078 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:01.309389 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:01.312377 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-1/data/instance:
uuid: "fbb592228e14496c96030eb115eb35c9"
format_stamp: "Formatted at 2026-06-20 02:52:01 on dist-test-slave-8p8c"
server_key: "29d8540cbf60649c92bc0798c53be69f"
server_key_iv: "c1efec3a018194b8d77b4fb604760fa1"
server_key_version: "encryptionkey@0"
I20260620 02:52:01.313182 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-1/wal/instance:
uuid: "fbb592228e14496c96030eb115eb35c9"
format_stamp: "Formatted at 2026-06-20 02:52:01 on dist-test-slave-8p8c"
server_key: "29d8540cbf60649c92bc0798c53be69f"
server_key_iv: "c1efec3a018194b8d77b4fb604760fa1"
server_key_version: "encryptionkey@0"
I20260620 02:52:01.318589 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 02:52:01.322449 1174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:01.323865 1159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:52:01.324075 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-1/wal
uuid: "fbb592228e14496c96030eb115eb35c9"
format_stamp: "Formatted at 2026-06-20 02:52:01 on dist-test-slave-8p8c"
server_key: "29d8540cbf60649c92bc0798c53be69f"
server_key_iv: "c1efec3a018194b8d77b4fb604760fa1"
server_key_version: "encryptionkey@0"
I20260620 02:52:01.324285 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:01.339324 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:01.343850 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:01.344208 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:01.345191 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:52:01.346725 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:01.346824 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:01.346971 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:01.347056 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:01.381073 1159 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39899
I20260620 02:52:01.381114 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39899 every 8 connection(s)
I20260620 02:52:01.382428 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:52:01.385301 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 1159
I20260620 02:52:01.385526 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:52:01.386024 388 external_mini_cluster.cc:1428] Setting key 03f27e26954a4eb6b8962db2ef11ccb5
Jun 20 02:52:01 dist-test-slave-8p8c krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 02:52:01.407812 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:01.385457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46649 (local address 127.0.97.62:38129)
0620 02:52:01.388193 (+ 2736us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:01.388198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:01.388217 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:52:01.390446 (+ 2229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:01.390452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:01.390532 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:01.390635 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:01.392995 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:01.394068 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:01.396011 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:01.396462 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:01.396611 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:01.400239 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:01.400249 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:01.400255 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:01.400306 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:01.404311 (+ 4005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:01.405198 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:01.405202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:01.405205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:01.405324 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:01.405828 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:01.405831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:01.405834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:01.406114 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:01.406237 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:01.407146 (+ 909us) server_negotiation.cc:299] Negotiation successful
0620 02:52:01.407409 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2619,"thread_start_us":2517,"threads_started":1}
I20260620 02:52:01.409377 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:01.386210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38129 (local address 127.0.97.2:46649)
0620 02:52:01.387354 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:01.387418 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:01.389990 (+ 2572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:01.391027 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:01.391043 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:01.391892 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:01.392785 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:01.392813 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:01.394248 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:01.394252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:01.395822 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:01.395833 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:01.396702 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:01.396708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:01.397041 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:01.397758 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:01.397781 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:01.399969 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:01.404490 (+ 4521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:01.404498 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:01.404513 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:01.404986 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:01.405459 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:01.405463 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:01.405466 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:01.405681 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:01.406334 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:01.406343 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:01.406757 (+ 414us) client_negotiation.cc:769] Sending connection context
0620 02:52:01.407042 (+ 285us) client_negotiation.cc:240] Negotiation successful
0620 02:52:01.407648 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":422,"threads_started":1}
I20260620 02:52:01.411171 1287 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38129
I20260620 02:52:01.411514 1287 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:01.412277 1287 heartbeater.cc:511] Master 127.0.97.62:38129 requested a full tablet report, sending...
I20260620 02:52:01.413818 964 ts_manager.cc:194] Registered new tserver with Master: fbb592228e14496c96030eb115eb35c9 (127.0.97.2:39899)
I20260620 02:52:01.415150 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46649
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:52:01.435169 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38129
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:01.754340 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:01.754683 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:01.754779 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:01.761842 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:01.761947 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:01.762059 1293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:52:01.771687 1293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38129
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:01.772887 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:01.774142 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:01.787408 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:01.787472 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:01.787497 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:01.788316 1293 server_base.cc:1034] running on GCE node
I20260620 02:52:01.788897 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:01.789794 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:01.791038 1293 hybrid_clock.cc:648] HybridClock initialized: now 1781923921790985 us; error 59 us; skew 500 ppm
I20260620 02:52:01.791337 1293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:01 dist-test-slave-8p8c krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:01.794579 1293 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:52:01.796005 1293 webserver.cc:460] Webserver started at http://127.0.97.3:38999/ using document root <none> and password file <none>
I20260620 02:52:01.796687 1293 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:01.796833 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:01.797147 1293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:01.800086 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-2/data/instance:
uuid: "84362b57063244e58164b07ad13c20ad"
format_stamp: "Formatted at 2026-06-20 02:52:01 on dist-test-slave-8p8c"
server_key: "15c145619d9f4b519d9a40cac61577a7"
server_key_iv: "d0509622e7954267f9b660820aae09f8"
server_key_version: "encryptionkey@0"
I20260620 02:52:01.800858 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-2/wal/instance:
uuid: "84362b57063244e58164b07ad13c20ad"
format_stamp: "Formatted at 2026-06-20 02:52:01 on dist-test-slave-8p8c"
server_key: "15c145619d9f4b519d9a40cac61577a7"
server_key_iv: "d0509622e7954267f9b660820aae09f8"
server_key_version: "encryptionkey@0"
I20260620 02:52:01.805864 1293 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 02:52:01.809307 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:01.810698 1293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 02:52:01.810883 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-2/wal
uuid: "84362b57063244e58164b07ad13c20ad"
format_stamp: "Formatted at 2026-06-20 02:52:01 on dist-test-slave-8p8c"
server_key: "15c145619d9f4b519d9a40cac61577a7"
server_key_iv: "d0509622e7954267f9b660820aae09f8"
server_key_version: "encryptionkey@0"
I20260620 02:52:01.811100 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:01.823314 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:01.827657 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:01.827989 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:01.828998 1293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:52:01.830345 1293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:01.830482 1293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:01.830633 1293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:01.830705 1293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:01.865497 1293 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37101
I20260620 02:52:01.865531 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37101 every 8 connection(s)
I20260620 02:52:01.866882 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:52:01.872021 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 1293
I20260620 02:52:01.872215 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781923914418765-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:52:01.872722 388 external_mini_cluster.cc:1428] Setting key 3feb6f4bb7b5617bb7b06ae0ec3f5d8d
Jun 20 02:52:01 dist-test-slave-8p8c krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:01.885617 1153 heartbeater.cc:503] Master 127.0.97.62:38129 was elected leader, sending a full tablet report...
I20260620 02:52:01.889232 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:01.869546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36953 (local address 127.0.97.62:38129)
0620 02:52:01.869755 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:01.869762 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:01.871089 (+ 1327us) server_negotiation.cc:407] Connection header received
0620 02:52:01.873491 (+ 2402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:01.873499 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:01.873603 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:01.873689 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:01.875613 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:01.876391 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:01.877929 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:01.878314 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:01.878528 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:01.882715 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:01.882720 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:01.882724 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:01.882756 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:01.885755 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:01.886733 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:01.886738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:01.886742 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:01.886890 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:01.887398 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:01.887403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:01.887407 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:01.887776 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:01.887937 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:01.888699 (+ 762us) server_negotiation.cc:299] Negotiation successful
0620 02:52:01.888915 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:52:01.890244 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:01.870131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38129 (local address 127.0.97.3:36953)
0620 02:52:01.870918 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:01.870960 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:01.873168 (+ 2208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:01.873971 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:01.873986 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:01.874810 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:01.875433 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:01.875457 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:01.876546 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:01.876550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:01.877784 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:01.877799 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:01.878612 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:01.878617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:01.879250 (+ 633us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:01.879890 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:01.879913 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:01.882498 (+ 2585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:01.885928 (+ 3430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:01.885938 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:01.885954 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:01.886607 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:01.887019 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:01.887024 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:01.887028 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:01.887268 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:01.887911 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:01.887919 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:01.888345 (+ 426us) client_negotiation.cc:769] Sending connection context
0620 02:52:01.888603 (+ 258us) client_negotiation.cc:240] Negotiation successful
0620 02:52:01.889130 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":364,"threads_started":1}
I20260620 02:52:01.891577 1421 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38129
I20260620 02:52:01.891872 1421 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:01.892529 1421 heartbeater.cc:511] Master 127.0.97.62:38129 requested a full tablet report, sending...
I20260620 02:52:01.893930 964 ts_manager.cc:194] Registered new tserver with Master: 84362b57063244e58164b07ad13c20ad (127.0.97.3:37101)
I20260620 02:52:01.894726 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36953
I20260620 02:52:01.900532 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:52:01.920024 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:01.906513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52852 (local address 127.0.97.62:38129)
0620 02:52:01.906693 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:01.906697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:01.907666 (+ 969us) server_negotiation.cc:407] Connection header received
0620 02:52:01.907792 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:01.907795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:01.907856 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:01.907937 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:01.909340 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:01.910058 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:01.911554 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:01.911967 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:01.912088 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:01.914501 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:01.914507 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:01.914510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:01.914542 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:01.917340 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:01.918006 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:01.918011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:01.918013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:01.918142 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:01.918606 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:01.918610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:01.918612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:01.918946 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:01.919131 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:01.919446 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 02:52:01.919618 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 02:52:01.926661 964 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:52:01.933995 964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:52:01.962770 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:01.946846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44061 (local address 127.0.0.1:42324)
0620 02:52:01.949594 (+ 2748us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:01.949613 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:01.949739 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:01.951324 (+ 1585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:01.951335 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:01.951377 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:01.951882 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:01.951899 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:01.954636 (+ 2737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:01.954644 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:01.958104 (+ 3460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:01.958126 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:01.961201 (+ 3075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:01.961211 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:01.961566 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:01.961626 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 02:52:01.961781 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 02:52:01.962299 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2620,"mutex_wait_us":1,"thread_start_us":2441,"threads_started":1}
I20260620 02:52:01.962857 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:01.947691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42324 (local address 127.0.97.1:44061)
0620 02:52:01.949301 (+ 1610us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:01.949312 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:01.949724 (+ 412us) server_negotiation.cc:407] Connection header received
0620 02:52:01.949931 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:01.949941 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:01.950740 (+ 799us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:01.950980 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:01.952113 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:01.954399 (+ 2286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:01.958414 (+ 4015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:01.960527 (+ 2113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:01.960697 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:01.961020 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:01.962105 (+ 1085us) server_negotiation.cc:299] Negotiation successful
0620 02:52:01.962351 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1381,"thread_start_us":1203,"threads_started":1}
I20260620 02:52:01.964787 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:01.946735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48078 (local address 127.0.97.3:37101)
0620 02:52:01.949788 (+ 3053us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:01.949800 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:01.949836 (+ 36us) server_negotiation.cc:407] Connection header received
0620 02:52:01.949956 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:01.949968 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:01.950265 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:01.950534 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:01.953547 (+ 3013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:01.957016 (+ 3469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:01.962133 (+ 5117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:01.963680 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:01.963811 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:01.964040 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:01.964218 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 02:52:01.964411 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2889,"thread_start_us":2738,"threads_started":1}
I20260620 02:52:01.965682 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:01.946347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37101 (local address 127.0.0.1:48078)
0620 02:52:01.947421 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:01.947463 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:01.947645 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:01.952533 (+ 4888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:01.952540 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:01.952576 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:01.953103 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:01.953118 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:01.957225 (+ 4107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:01.957233 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:01.961945 (+ 4712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:01.961960 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:01.963858 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:01.963864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:01.964029 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:01.964061 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 02:52:01.964151 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 02:52:01.964280 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"mutex_wait_us":444,"thread_start_us":646,"threads_started":1}
I20260620 02:52:01.968433 1088 tablet_service.cc:1467] Processing CreateTablet for tablet a66b216deb4f4935b4a583ab77d7b34d (DEFAULT_TABLE table=test-table [id=6e1439a542c84d77992c5e2177a0dd14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:01.970175 1088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a66b216deb4f4935b4a583ab77d7b34d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:01.969892 1356 tablet_service.cc:1467] Processing CreateTablet for tablet a66b216deb4f4935b4a583ab77d7b34d (DEFAULT_TABLE table=test-table [id=6e1439a542c84d77992c5e2177a0dd14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:01.971149 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:01.950098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44568 (local address 127.0.97.2:39899)
0620 02:52:01.961940 (+ 11842us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:01.961954 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:01.963637 (+ 1683us) server_negotiation.cc:407] Connection header received
0620 02:52:01.963799 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:01.963808 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:01.964149 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:01.964404 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:01.965184 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:01.966815 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:01.968761 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:01.970054 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:01.970224 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:01.970494 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:01.970675 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 02:52:01.970836 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11627,"thread_start_us":11492,"threads_started":1}
I20260620 02:52:01.971146 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:01.949706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39899 (local address 127.0.0.1:44568)
0620 02:52:01.963495 (+ 13789us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:01.963513 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:01.963647 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:01.964652 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:01.964657 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:01.964683 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:01.964986 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:01.964996 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:01.967032 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:01.967040 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:01.968602 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:01.968619 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:01.970272 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:01.970279 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:01.970474 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:01.970510 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 02:52:01.970636 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 02:52:01.970780 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13647,"thread_start_us":13525,"threads_started":1}
I20260620 02:52:01.971791 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a66b216deb4f4935b4a583ab77d7b34d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:01.975183 1222 tablet_service.cc:1467] Processing CreateTablet for tablet a66b216deb4f4935b4a583ab77d7b34d (DEFAULT_TABLE table=test-table [id=6e1439a542c84d77992c5e2177a0dd14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:01.976634 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a66b216deb4f4935b4a583ab77d7b34d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:01.986274 1439 tablet_bootstrap.cc:492] T a66b216deb4f4935b4a583ab77d7b34d P 9a6977eae9034f1a9e290689255669f7: Bootstrap starting.
I20260620 02:52:01.989200 1440 tablet_bootstrap.cc:492] T a66b216deb4f4935b4a583ab77d7b34d P 84362b57063244e58164b07ad13c20ad: Bootstrap starting.
I20260620 02:52:01.990954 1439 tablet_bootstrap.cc:654] T a66b216deb4f4935b4a583ab77d7b34d P 9a6977eae9034f1a9e290689255669f7: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:01.991524 1441 tablet_bootstrap.cc:492] T a66b216deb4f4935b4a583ab77d7b34d P fbb592228e14496c96030eb115eb35c9: Bootstrap starting.
I20260620 02:52:01.992383 1439 log.cc:824] T a66b216deb4f4935b4a583ab77d7b34d P 9a6977eae9034f1a9e290689255669f7: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:01.992785 1440 tablet_bootstrap.cc:654] T a66b216deb4f4935b4a583ab77d7b34d P 84362b57063244e58164b07ad13c20ad: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:01.994046 1440 log.cc:824] T a66b216deb4f4935b4a583ab77d7b34d P 84362b57063244e58164b07ad13c20ad: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:01.995414 1441 tablet_bootstrap.cc:654] T a66b216deb4f4935b4a583ab77d7b34d P fbb592228e14496c96030eb115eb35c9: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:01.996150 1439 tablet_bootstrap.cc:492] T a66b216deb4f4935b4a583ab77d7b34d P 9a6977eae9034f1a9e290689255669f7: No bootstrap required, opened a new log
I20260620 02:52:01.996417 1441 log.cc:824] T a66b216deb4f4935b4a583ab77d7b34d P fbb592228e14496c96030eb115eb35c9: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:01.996477 1439 ts_tablet_manager.cc:1397] T a66b216deb4f4935b4a583ab77d7b34d P 9a6977eae9034f1a9e290689255669f7: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260620 02:52:01.997360 1440 tablet_bootstrap.cc:492] T a66b216deb4f4935b4a583ab77d7b34d P 84362b57063244e58164b07ad13c20ad: No bootstrap required, opened a new log
I20260620 02:52:01.997624 1440 ts_tablet_manager.cc:1397] T a66b216deb4f4935b4a583ab77d7b34d P 84362b57063244e58164b07ad13c20ad: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 02:52:01.999194 1441 tablet_bootstrap.cc:492] T a66b216deb4f4935b4a583ab77d7b34d P fbb592228e14496c96030eb115eb35c9: No bootstrap required, opened a new log
I20260620 02:52:01.999480 1441 ts_tablet_manager.cc:1397] T a66b216deb4f4935b4a583ab77d7b34d P fbb592228e14496c96030eb115eb35c9: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 02:52:02.007519 1441 raft_consensus.cc:348] T a66b216deb4f4935b4a583ab77d7b34d P fbb592228e14496c96030eb115eb35c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84362b57063244e58164b07ad13c20ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37101 } } peers { permanent_uuid: "9a6977eae9034f1a9e290689255669f7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44061 } } peers { permanent_uuid: "fbb592228e14496c96030eb115eb35c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39899 } }
I20260620 02:52:02.008209 1441 raft_consensus.cc:374] T a66b216deb4f4935b4a583ab77d7b34d P fbb592228e14496c96030eb115eb35c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:02.008352 1441 raft_consensus.cc:729] T a66b216deb4f4935b4a583ab77d7b34d P fbb592228e14496c96030eb115eb35c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb592228e14496c96030eb115eb35c9, State: Initialized, Role: FOLLOWER
I20260620 02:52:02.008380 1439 raft_consensus.cc:348] T a66b216deb4f4935b4a583ab77d7b34d P 9a6977eae9034f1a9e290689255669f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84362b57063244e58164b07ad13c20ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37101 } } peers { permanent_uuid: "9a6977eae9034f1a9e290689255669f7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44061 } } peers { permanent_uuid: "fbb592228e14496c96030eb115eb35c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39899 } }
I20260620 02:52:02.008895 1439 raft_consensus.cc:374] T a66b216deb4f4935b4a583ab77d7b34d P 9a6977eae9034f1a9e290689255669f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:02.009021 1439 raft_consensus.cc:729] T a66b216deb4f4935b4a583ab77d7b34d P 9a6977eae9034f1a9e290689255669f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a6977eae9034f1a9e290689255669f7, State: Initialized, Role: FOLLOWER
I20260620 02:52:02.009173 1441 consensus_queue.cc:260] T a66b216deb4f4935b4a583ab77d7b34d P fbb592228e14496c96030eb115eb35c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84362b57063244e58164b07ad13c20ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37101 } } peers { permanent_uuid: "9a6977eae9034f1a9e290689255669f7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44061 } } peers { permanent_uuid: "fbb592228e14496c96030eb115eb35c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39899 } }
I20260620 02:52:02.009002 1440 raft_consensus.cc:348] T a66b216deb4f4935b4a583ab77d7b34d P 84362b57063244e58164b07ad13c20ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84362b57063244e58164b07ad13c20ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37101 } } peers { permanent_uuid: "9a6977eae9034f1a9e290689255669f7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44061 } } peers { permanent_uuid: "fbb592228e14496c96030eb115eb35c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39899 } }
I20260620 02:52:02.009550 1440 raft_consensus.cc:374] T a66b216deb4f4935b4a583ab77d7b34d P 84362b57063244e58164b07ad13c20ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:02.009660 1440 raft_consensus.cc:729] T a66b216deb4f4935b4a583ab77d7b34d P 84362b57063244e58164b07ad13c20ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84362b57063244e58164b07ad13c20ad, State: Initialized, Role: FOLLOWER
I20260620 02:52:02.009747 1439 consensus_queue.cc:260] T a66b216deb4f4935b4a583ab77d7b34d P 9a6977eae9034f1a9e290689255669f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84362b57063244e58164b07ad13c20ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37101 } } peers { permanent_uuid: "9a6977eae9034f1a9e290689255669f7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44061 } } peers { permanent_uuid: "fbb592228e14496c96030eb115eb35c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39899 } }
I20260620 02:52:02.010269 1440 consensus_queue.cc:260] T a66b216deb4f4935b4a583ab77d7b34d P 84362b57063244e58164b07ad13c20ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84362b57063244e58164b07ad13c20ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37101 } } peers { permanent_uuid: "9a6977eae9034f1a9e290689255669f7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44061 } } peers { permanent_uuid: "fbb592228e14496c96030eb115eb35c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39899 } }
I20260620 02:52:02.010540 1287 heartbeater.cc:503] Master 127.0.97.62:38129 was elected leader, sending a full tablet report...
I20260620 02:52:02.013676 1439 ts_tablet_manager.cc:1428] T a66b216deb4f4935b4a583ab77d7b34d P 9a6977eae9034f1a9e290689255669f7: Time spent starting tablet: real 0.017s user 0.012s sys 0.002s
I20260620 02:52:02.014816 1421 heartbeater.cc:503] Master 127.0.97.62:38129 was elected leader, sending a full tablet report...
I20260620 02:52:02.016809 1440 ts_tablet_manager.cc:1428] T a66b216deb4f4935b4a583ab77d7b34d P 84362b57063244e58164b07ad13c20ad: Time spent starting tablet: real 0.019s user 0.019s sys 0.000s
I20260620 02:52:02.016970 1441 ts_tablet_manager.cc:1428] T a66b216deb4f4935b4a583ab77d7b34d P fbb592228e14496c96030eb115eb35c9: Time spent starting tablet: real 0.017s user 0.006s sys 0.005s
W20260620 02:52:02.103842 1154 tablet.cc:2369] T a66b216deb4f4935b4a583ab77d7b34d P 9a6977eae9034f1a9e290689255669f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:52:02.118649 1422 tablet.cc:2369] T a66b216deb4f4935b4a583ab77d7b34d P 84362b57063244e58164b07ad13c20ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:52:02.127713 1445 raft_consensus.cc:482] T a66b216deb4f4935b4a583ab77d7b34d P fbb592228e14496c96030eb115eb35c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:52:02.128005 1445 raft_consensus.cc:504] T a66b216deb4f4935b4a583ab77d7b34d P fbb592228e14496c96030eb115eb35c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84362b57063244e58164b07ad13c20ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37101 } } peers { permanent_uuid: "9a6977eae9034f1a9e290689255669f7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44061 } } peers { permanent_uuid: "fbb592228e14496c96030eb115eb35c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39899 } }
I20260620 02:52:02.129830 1445 leader_election.cc:288] T a66b216deb4f4935b4a583ab77d7b34d P fbb592228e14496c96030eb115eb35c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84362b57063244e58164b07ad13c20ad (127.0.97.3:37101), 9a6977eae9034f1a9e290689255669f7 (127.0.97.1:44061)
W20260620 02:52:02.134568 1288 tablet.cc:2369] T a66b216deb4f4935b4a583ab77d7b34d P fbb592228e14496c96030eb115eb35c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:52:02.139469 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:02.130443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54817 (local address 127.0.97.1:44061)
0620 02:52:02.130723 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:02.130732 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:02.131379 (+ 647us) server_negotiation.cc:407] Connection header received
0620 02:52:02.131614 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:02.131623 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:02.131698 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:02.131792 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:02.132908 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:02.133966 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:02.137043 (+ 3077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:02.138286 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:02.138440 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:02.138556 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:02.139090 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 02:52:02.139221 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 02:52:02.139464 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:02.130215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47931 (local address 127.0.97.3:37101)
0620 02:52:02.130500 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:02.130509 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:02.131212 (+ 703us) server_negotiation.cc:407] Connection header received
0620 02:52:02.131480 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:02.131484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:02.131543 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:02.131646 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:02.132749 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:02.133967 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:02.137081 (+ 3114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:02.138284 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:02.138440 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:02.138545 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:02.139115 (+ 570us) server_negotiation.cc:299] Negotiation successful
0620 02:52:02.139235 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:52:02.139643 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:02.130155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44061 (local address 127.0.97.2:54817)
0620 02:52:02.131283 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:02.131296 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:02.131419 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:02.131829 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:02.131834 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:02.132265 (+ 431us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:02.132753 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:02.132762 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:02.134108 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:02.134112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:02.136873 (+ 2761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:02.136887 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:02.138565 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:02.138574 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:02.138868 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:02.138946 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 02:52:02.139041 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 02:52:02.139185 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1033,"mutex_wait_us":254,"thread_start_us":784,"threads_started":1}
I20260620 02:52:02.139909 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:02.130144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37101 (local address 127.0.97.2:47931)
0620 02:52:02.131097 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:02.131116 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:02.131290 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:02.131660 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:02.131664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:02.131686 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:02.132574 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:02.132588 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:02.134107 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:02.134112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:02.136874 (+ 2762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:02.136887 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:02.138608 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:02.138614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:02.138864 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:02.138942 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 02:52:02.139041 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 02:52:02.139185 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":199,"thread_start_us":578,"threads_started":1}
I20260620 02:52:02.140750 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a66b216deb4f4935b4a583ab77d7b34d" candidate_uuid: "fbb592228e14496c96030eb115eb35c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a6977eae9034f1a9e290689255669f7" is_pre_election: true
I20260620 02:52:02.140833 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a66b216deb4f4935b4a583ab77d7b34d" candidate_uuid: "fbb592228e14496c96030eb115eb35c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84362b57063244e58164b07ad13c20ad" is_pre_election: true
I20260620 02:52:02.141275 1108 raft_consensus.cc:2454] T a66b216deb4f4935b4a583ab77d7b34d P 9a6977eae9034f1a9e290689255669f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbb592228e14496c96030eb115eb35c9 in term 0.
I20260620 02:52:02.141294 1376 raft_consensus.cc:2454] T a66b216deb4f4935b4a583ab77d7b34d P 84362b57063244e58164b07ad13c20ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbb592228e14496c96030eb115eb35c9 in term 0.
I20260620 02:52:02.141916 1176 leader_election.cc:302] T a66b216deb4f4935b4a583ab77d7b34d P fbb592228e14496c96030eb115eb35c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a6977eae9034f1a9e290689255669f7, fbb592228e14496c96030eb115eb35c9; no voters:
I20260620 02:52:02.142306 1445 raft_consensus.cc:2781] T a66b216deb4f4935b4a583ab77d7b34d P fbb592228e14496c96030eb115eb35c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:52:02.142927 1445 raft_consensus.cc:482] T a66b216deb4f4935b4a583ab77d7b34d P fbb592228e14496c96030eb115eb35c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:52:02.143035 1445 raft_consensus.cc:3037] T a66b216deb4f4935b4a583ab77d7b34d P fbb592228e14496c96030eb115eb35c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:02.145519 1445 raft_consensus.cc:504] T a66b216deb4f4935b4a583ab77d7b34d P fbb592228e14496c96030eb115eb35c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84362b57063244e58164b07ad13c20ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37101 } } peers { permanent_uuid: "9a6977eae9034f1a9e290689255669f7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44061 } } peers { permanent_uuid: "fbb592228e14496c96030eb115eb35c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39899 } }
I20260620 02:52:02.146162 1445 leader_election.cc:288] T a66b216deb4f4935b4a583ab77d7b34d P fbb592228e14496c96030eb115eb35c9 [CANDIDATE]: Term 1 election: Requested vote from peers 84362b57063244e58164b07ad13c20ad (127.0.97.3:37101), 9a6977eae9034f1a9e290689255669f7 (127.0.97.1:44061)
I20260620 02:52:02.146610 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a66b216deb4f4935b4a583ab77d7b34d" candidate_uuid: "fbb592228e14496c96030eb115eb35c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84362b57063244e58164b07ad13c20ad"
I20260620 02:52:02.146683 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a66b216deb4f4935b4a583ab77d7b34d" candidate_uuid: "fbb592228e14496c96030eb115eb35c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a6977eae9034f1a9e290689255669f7"
I20260620 02:52:02.146847 1376 raft_consensus.cc:3037] T a66b216deb4f4935b4a583ab77d7b34d P 84362b57063244e58164b07ad13c20ad [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:02.146904 1108 raft_consensus.cc:3037] T a66b216deb4f4935b4a583ab77d7b34d P 9a6977eae9034f1a9e290689255669f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:02.149122 1108 raft_consensus.cc:2454] T a66b216deb4f4935b4a583ab77d7b34d P 9a6977eae9034f1a9e290689255669f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbb592228e14496c96030eb115eb35c9 in term 1.
I20260620 02:52:02.149540 1176 leader_election.cc:302] T a66b216deb4f4935b4a583ab77d7b34d P fbb592228e14496c96030eb115eb35c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a6977eae9034f1a9e290689255669f7, fbb592228e14496c96030eb115eb35c9; no voters:
I20260620 02:52:02.149847 1445 raft_consensus.cc:2781] T a66b216deb4f4935b4a583ab77d7b34d P fbb592228e14496c96030eb115eb35c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:02.150050 1376 raft_consensus.cc:2454] T a66b216deb4f4935b4a583ab77d7b34d P 84362b57063244e58164b07ad13c20ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbb592228e14496c96030eb115eb35c9 in term 1.
I20260620 02:52:02.150408 1445 raft_consensus.cc:686] T a66b216deb4f4935b4a583ab77d7b34d P fbb592228e14496c96030eb115eb35c9 [term 1 LEADER]: Becoming Leader. State: Replica: fbb592228e14496c96030eb115eb35c9, State: Running, Role: LEADER
I20260620 02:52:02.150951 1445 consensus_queue.cc:237] T a66b216deb4f4935b4a583ab77d7b34d P fbb592228e14496c96030eb115eb35c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84362b57063244e58164b07ad13c20ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37101 } } peers { permanent_uuid: "9a6977eae9034f1a9e290689255669f7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44061 } } peers { permanent_uuid: "fbb592228e14496c96030eb115eb35c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39899 } }
I20260620 02:52:02.155318 962 catalog_manager.cc:5495] T a66b216deb4f4935b4a583ab77d7b34d P fbb592228e14496c96030eb115eb35c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbb592228e14496c96030eb115eb35c9 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "fbb592228e14496c96030eb115eb35c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84362b57063244e58164b07ad13c20ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a6977eae9034f1a9e290689255669f7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbb592228e14496c96030eb115eb35c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39899 } health_report { overall_health: HEALTHY } } }
Jun 20 02:52:02 dist-test-slave-8p8c krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:52:02.208372 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:02.193242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37672 (local address 127.0.97.1:44061)
0620 02:52:02.193455 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:02.193465 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:02.193959 (+ 494us) server_negotiation.cc:407] Connection header received
0620 02:52:02.194108 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:02.194112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:02.194189 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:02.194273 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:02.195834 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:02.196602 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:02.197847 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:02.198269 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:02.198386 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:02.202793 (+ 4407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:02.202815 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:02.202832 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:02.202874 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:02.205810 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:02.206545 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:02.206552 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:02.206555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:02.206717 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:02.207048 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:02.207051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:02.207053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:02.207402 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:02.207627 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:02.207915 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 02:52:02.208110 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Jun 20 02:52:02 dist-test-slave-8p8c krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923922, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 02:52:02 dist-test-slave-8p8c krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923922, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:52:02.253420 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:02.239796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37676 (local address 127.0.97.1:44061)
0620 02:52:02.240044 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:02.240050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:02.240313 (+ 263us) server_negotiation.cc:407] Connection header received
0620 02:52:02.240453 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:02.240457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:02.240521 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:02.240610 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:02.241874 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:02.242664 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:02.243882 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:02.244296 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:02.244399 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:02.248211 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:02.248216 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:02.248219 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:02.248254 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:02.250889 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:02.251435 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:02.251438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:02.251441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:02.251558 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:02.252189 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:02.252192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:02.252194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:02.252438 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:02.252646 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:02.252875 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 02:52:02.253067 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 02:52:02 dist-test-slave-8p8c krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:52:02 dist-test-slave-8p8c krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:52:02.298781 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:02.284387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37682 (local address 127.0.97.1:44061)
0620 02:52:02.284597 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:02.284605 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:02.284978 (+ 373us) server_negotiation.cc:407] Connection header received
0620 02:52:02.285193 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:02.285197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:02.285259 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:02.285370 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:02.286792 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:02.287471 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:02.288711 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:02.289144 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:02.289273 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:02.293680 (+ 4407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:02.293687 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:02.293690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:02.293732 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:02.296295 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:02.296937 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:02.296943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:02.296947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:02.297073 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:02.297380 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:02.297383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:02.297385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:02.297690 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:02.297872 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:02.298141 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 02:52:02.298322 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 02:52:02.300673 1068 tablet_service.cc:2908] Rejecting scan request for tablet a66b216deb4f4935b4a583ab77d7b34d: Uninitialized: clean time has not yet been initialized
I20260620 02:52:02.303738 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 1025
I20260620 02:52:02.317159 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:02.305254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37688 (local address 127.0.97.1:44061)
0620 02:52:02.305403 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:02.305408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:02.305803 (+ 395us) server_negotiation.cc:407] Connection header received
0620 02:52:02.305924 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:02.305927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:02.306012 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:02.306094 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:02.307271 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:02.307965 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:02.309226 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:02.309675 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:02.309786 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:02.312047 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:02.312052 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:02.312055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:02.312083 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:02.314670 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:02.315334 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:02.315338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:02.315341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:02.315477 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:02.315821 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:02.315825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:02.315827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:02.316126 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:02.316325 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:02.316768 (+ 443us) server_negotiation.cc:299] Negotiation successful
0620 02:52:02.316939 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 02:52:02.317786 1148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:02.408516 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 1025
I20260620 02:52:02.425388 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 1159
Jun 20 02:52:02 dist-test-slave-8p8c krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:52:02.443315 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:02.426515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45432 (local address 127.0.97.2:39899)
0620 02:52:02.426744 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:02.426750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:02.426820 (+ 70us) server_negotiation.cc:407] Connection header received
0620 02:52:02.427049 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:02.427055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:02.427131 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:02.427247 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:02.428752 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:02.429503 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:02.431197 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:02.431807 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:02.431952 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:02.437071 (+ 5119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:02.437087 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:02.437104 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:02.437141 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:02.440146 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:02.440855 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:02.440863 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:02.440865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:02.441022 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:02.441422 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:02.441431 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:02.441434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:02.441755 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:02.442074 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:02.442300 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 02:52:02.442583 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 02:52:02.444059 1282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:02.534401 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 1159
I20260620 02:52:02.551054 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 1293
Jun 20 02:52:02 dist-test-slave-8p8c krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:52:02.566350 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:02.551853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45964 (local address 127.0.97.3:37101)
0620 02:52:02.552035 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:02.552040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:02.552058 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:52:02.552243 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:02.552247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:02.552323 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:02.552410 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:02.553663 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:02.554447 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:02.555757 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:02.556179 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:02.556284 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:02.560587 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:02.560601 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:02.560622 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:02.560660 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:02.563669 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:02.564359 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:02.564367 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:02.564369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:02.564519 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:02.564971 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:02.564975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:02.564977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:02.565337 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:02.565553 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:02.565874 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 02:52:02.566094 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 02:52:02.567130 1416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:02.655958 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 1293
I20260620 02:52:02.673003 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 932
I20260620 02:52:02.674268 994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:02.783864 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 932
2026-06-20T02:52:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4260 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:52:02 dist-test-slave-8p8c krb5kdc[1469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:52:02 dist-test-slave-8p8c krb5kdc[1469](info): set up 2 sockets
Jun 20 02:52:02 dist-test-slave-8p8c krb5kdc[1469](info): commencing operation
krb5kdc: starting...
W20260620 02:52:03.886456 388 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:52:04 dist-test-slave-8p8c krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:52:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:52:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:52:04.083448 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38391
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:04.401844 1482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:04.402135 1482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:04.402287 1482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:04.409641 1482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:52:04.409754 1482 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 02:52:04.409825 1482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:04.409927 1482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:04.410023 1482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:52:04.410112 1482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:52:04.420032 1482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38391
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:04.421357 1482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:04.422771 1482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:04.437317 1487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:04.439939 1488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:04.441414 1490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:04.442171 1482 server_base.cc:1034] running on GCE node
I20260620 02:52:04.443133 1482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:04.445263 1482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:04.446482 1482 hybrid_clock.cc:648] HybridClock initialized: now 1781923924446385 us; error 67 us; skew 500 ppm
I20260620 02:52:04.446883 1482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:04 dist-test-slave-8p8c krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:04.449997 1482 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:52:04.451654 1482 webserver.cc:460] Webserver started at http://127.0.97.62:43781/ using document root <none> and password file <none>
I20260620 02:52:04.452422 1482 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:04.452550 1482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:04.452886 1482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:04.455955 1482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "32de122f2423469682096ed5bb608fa8"
format_stamp: "Formatted at 2026-06-20 02:52:04 on dist-test-slave-8p8c"
server_key: "e527f45a5e3a41e4bd2903c04c828d1f"
server_key_iv: "b2cc4ac719ce76069f74b21d22609542"
server_key_version: "encryptionkey@0"
I20260620 02:52:04.456777 1482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "32de122f2423469682096ed5bb608fa8"
format_stamp: "Formatted at 2026-06-20 02:52:04 on dist-test-slave-8p8c"
server_key: "e527f45a5e3a41e4bd2903c04c828d1f"
server_key_iv: "b2cc4ac719ce76069f74b21d22609542"
server_key_version: "encryptionkey@0"
I20260620 02:52:04.462096 1482 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 02:52:04.465863 1497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:04.467293 1482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:52:04.467484 1482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "32de122f2423469682096ed5bb608fa8"
format_stamp: "Formatted at 2026-06-20 02:52:04 on dist-test-slave-8p8c"
server_key: "e527f45a5e3a41e4bd2903c04c828d1f"
server_key_iv: "b2cc4ac719ce76069f74b21d22609542"
server_key_version: "encryptionkey@0"
I20260620 02:52:04.467710 1482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:04.491878 1482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:04.496609 1482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:04.496933 1482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:04.520727 1482 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38391
I20260620 02:52:04.520747 1558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38391 every 8 connection(s)
I20260620 02:52:04.522086 1482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:52:04.526124 1559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:04.531841 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 1482
I20260620 02:52:04.532084 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:52:04.532900 388 external_mini_cluster.cc:1428] Setting key cf0dde7074106bce970329ea66a8a735
I20260620 02:52:04.537992 1559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32de122f2423469682096ed5bb608fa8: Bootstrap starting.
I20260620 02:52:04.541229 1559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32de122f2423469682096ed5bb608fa8: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:04.542269 1559 log.cc:824] T 00000000000000000000000000000000 P 32de122f2423469682096ed5bb608fa8: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:04.545526 1559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32de122f2423469682096ed5bb608fa8: No bootstrap required, opened a new log
Jun 20 02:52:04 dist-test-slave-8p8c krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:04.553936 1559 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32de122f2423469682096ed5bb608fa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32de122f2423469682096ed5bb608fa8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38391 } }
I20260620 02:52:04.554333 1559 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32de122f2423469682096ed5bb608fa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:04.554515 1559 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32de122f2423469682096ed5bb608fa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32de122f2423469682096ed5bb608fa8, State: Initialized, Role: FOLLOWER
I20260620 02:52:04.555145 1559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32de122f2423469682096ed5bb608fa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32de122f2423469682096ed5bb608fa8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38391 } }
I20260620 02:52:04.555392 1559 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32de122f2423469682096ed5bb608fa8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:52:04.555538 1559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32de122f2423469682096ed5bb608fa8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:52:04.555729 1559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32de122f2423469682096ed5bb608fa8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:04.557318 1562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:04.535300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46216 (local address 127.0.97.62:38391)
0620 02:52:04.536263 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:04.536276 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:04.536346 (+ 70us) server_negotiation.cc:407] Connection header received
0620 02:52:04.538485 (+ 2139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:04.538511 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:04.538954 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:04.539389 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:04.540931 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:04.542449 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:04.544190 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:04.544922 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:04.545147 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:04.549483 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:04.549492 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:04.549509 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:04.549551 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:04.552955 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:04.553712 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:04.553725 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:04.553732 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:04.553889 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:04.554271 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:04.554278 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:04.554283 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:04.555060 (+ 777us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:04.555350 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:04.555829 (+ 479us) server_negotiation.cc:299] Negotiation successful
0620 02:52:04.556261 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":467,"threads_started":1}
I20260620 02:52:04.558537 1559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32de122f2423469682096ed5bb608fa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32de122f2423469682096ed5bb608fa8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38391 } }
I20260620 02:52:04.559152 1559 leader_election.cc:302] T 00000000000000000000000000000000 P 32de122f2423469682096ed5bb608fa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32de122f2423469682096ed5bb608fa8; no voters:
I20260620 02:52:04.559719 1559 leader_election.cc:288] T 00000000000000000000000000000000 P 32de122f2423469682096ed5bb608fa8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:52:04.559877 1564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32de122f2423469682096ed5bb608fa8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:04.561337 1564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32de122f2423469682096ed5bb608fa8 [term 1 LEADER]: Becoming Leader. State: Replica: 32de122f2423469682096ed5bb608fa8, State: Running, Role: LEADER
I20260620 02:52:04.561805 1559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32de122f2423469682096ed5bb608fa8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:52:04.561900 1564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32de122f2423469682096ed5bb608fa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32de122f2423469682096ed5bb608fa8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38391 } }
I20260620 02:52:04.565446 1565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32de122f2423469682096ed5bb608fa8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32de122f2423469682096ed5bb608fa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32de122f2423469682096ed5bb608fa8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38391 } } }
I20260620 02:52:04.565505 1566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32de122f2423469682096ed5bb608fa8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32de122f2423469682096ed5bb608fa8. Latest consensus state: current_term: 1 leader_uuid: "32de122f2423469682096ed5bb608fa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32de122f2423469682096ed5bb608fa8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38391 } } }
I20260620 02:52:04.565814 1565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32de122f2423469682096ed5bb608fa8 [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:04.565856 1566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32de122f2423469682096ed5bb608fa8 [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:04.566867 1572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:52:04.573383 1572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:52:04.582860 1572 catalog_manager.cc:1347] Generated new cluster ID: 8b399fefef834dfcb8a3b4100266ea33
I20260620 02:52:04.582990 1572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:52:04.610179 1572 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:52:04.612248 1572 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:52:04.625938 1572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32de122f2423469682096ed5bb608fa8: Generated new TSK 0
I20260620 02:52:04.627223 1572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:52:04.687387 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38391
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:05.018745 1585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:05.019047 1585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:05.019210 1585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:05.026644 1585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:05.026775 1585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:05.026922 1585 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:52:05.027045 1585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:52:05.036837 1585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:05.038192 1585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:05.039638 1585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:05.053239 1590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:05.053295 1591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:05.053256 1593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:05.054642 1585 server_base.cc:1034] running on GCE node
I20260620 02:52:05.055387 1585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:05.056411 1585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:05.057716 1585 hybrid_clock.cc:648] HybridClock initialized: now 1781923925057654 us; error 74 us; skew 500 ppm
I20260620 02:52:05.058058 1585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:05 dist-test-slave-8p8c krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923925, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:05.061348 1585 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:52:05.063100 1585 webserver.cc:460] Webserver started at http://127.0.97.1:42653/ using document root <none> and password file <none>
I20260620 02:52:05.063844 1585 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:05.063966 1585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:05.064302 1585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:05.067317 1585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-0/data/instance:
uuid: "173ed386c375473890ba5f27ebe29d22"
format_stamp: "Formatted at 2026-06-20 02:52:05 on dist-test-slave-8p8c"
server_key: "a6fa632aa8cd1728bc3dc71273103c2b"
server_key_iv: "6ac84cc542ba7790119cb20ddd0dcb81"
server_key_version: "encryptionkey@0"
I20260620 02:52:05.068207 1585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-0/wal/instance:
uuid: "173ed386c375473890ba5f27ebe29d22"
format_stamp: "Formatted at 2026-06-20 02:52:05 on dist-test-slave-8p8c"
server_key: "a6fa632aa8cd1728bc3dc71273103c2b"
server_key_iv: "6ac84cc542ba7790119cb20ddd0dcb81"
server_key_version: "encryptionkey@0"
I20260620 02:52:05.073530 1585 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 02:52:05.077082 1600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:05.078610 1585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:52:05.078790 1585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-0/wal
uuid: "173ed386c375473890ba5f27ebe29d22"
format_stamp: "Formatted at 2026-06-20 02:52:05 on dist-test-slave-8p8c"
server_key: "a6fa632aa8cd1728bc3dc71273103c2b"
server_key_iv: "6ac84cc542ba7790119cb20ddd0dcb81"
server_key_version: "encryptionkey@0"
I20260620 02:52:05.079020 1585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:05.109122 1585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:05.113803 1585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:05.114099 1585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:05.117023 1585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:05.117128 1585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:05.117259 1585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:05.117332 1585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:52:05 dist-test-slave-8p8c krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923925, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:05.143556 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:05.117818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50207 (local address 127.0.97.62:38391)
0620 02:52:05.118347 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:05.118352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:05.119610 (+ 1258us) server_negotiation.cc:407] Connection header received
0620 02:52:05.121809 (+ 2199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:05.121814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:05.121915 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:05.122038 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:05.125152 (+ 3114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:05.126080 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:05.129279 (+ 3199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:05.129844 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:05.130011 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:05.135473 (+ 5462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:05.135478 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:05.135482 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:05.135515 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:05.139299 (+ 3784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:05.140531 (+ 1232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:05.140540 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:05.140544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:05.140711 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:05.141295 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:05.141304 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:05.141310 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:05.141731 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:05.141926 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:05.142974 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0620 02:52:05.143194 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":312,"threads_started":1}
I20260620 02:52:05.144793 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:05.118519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38391 (local address 127.0.97.1:50207)
0620 02:52:05.119412 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:05.119463 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:05.121498 (+ 2035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:05.122513 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:05.122528 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:05.123666 (+ 1138us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:05.124914 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:05.124956 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:05.126248 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:05.126257 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:05.129066 (+ 2809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:05.129082 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:05.130139 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:05.130149 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:05.130486 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:05.131398 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:05.131450 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:05.135173 (+ 3723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:05.139472 (+ 4299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:05.139485 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:05.139507 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:05.140365 (+ 858us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:05.140819 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:05.140824 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:05.140834 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:05.141137 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:05.141903 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:05.141914 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:05.142463 (+ 549us) client_negotiation.cc:769] Sending connection context
0620 02:52:05.142880 (+ 417us) client_negotiation.cc:240] Negotiation successful
0620 02:52:05.143464 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":391,"threads_started":1}
I20260620 02:52:05.166497 1585 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40297
I20260620 02:52:05.166540 1720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40297 every 8 connection(s)
I20260620 02:52:05.168210 1585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:52:05.170820 1721 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38391
I20260620 02:52:05.171288 1721 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:05.171589 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:05.153320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33423 (local address 127.0.97.62:38391)
0620 02:52:05.153533 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:05.153541 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:05.155026 (+ 1485us) server_negotiation.cc:407] Connection header received
0620 02:52:05.155175 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:05.155183 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:05.155288 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:05.155421 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:05.156268 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:05.157446 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:05.159900 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:05.160665 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:05.160815 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:05.163726 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:05.163736 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:05.163744 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:05.163795 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:05.167488 (+ 3693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:05.168384 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:05.168391 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:05.168395 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:05.168551 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:05.168954 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:05.168962 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:05.168968 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:05.169479 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:05.170919 (+ 1440us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:05.171056 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 02:52:05.171279 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 02:52:05.172115 1721 heartbeater.cc:511] Master 127.0.97.62:38391 requested a full tablet report, sending...
I20260620 02:52:05.170985 1694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:05.153314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38391 (local address 127.0.97.1:33423)
0620 02:52:05.154913 (+ 1599us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:05.154931 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:05.155055 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:05.155444 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:05.155447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:05.155845 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:05.156125 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:05.156134 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:05.157620 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:05.157625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:05.159712 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:05.159733 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:05.160922 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:05.160932 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:05.161177 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:05.162042 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:05.162061 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:05.163474 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:05.167651 (+ 4177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:05.167655 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:05.167659 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:05.168207 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:05.168674 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:05.168681 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:05.168686 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:05.168826 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:05.169650 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:05.169655 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:05.169880 (+ 225us) client_negotiation.cc:769] Sending connection context
0620 02:52:05.170062 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 02:52:05.170464 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1451,"thread_start_us":1321,"threads_started":1}
I20260620 02:52:05.174558 1514 ts_manager.cc:194] Registered new tserver with Master: 173ed386c375473890ba5f27ebe29d22 (127.0.97.1:40297)
I20260620 02:52:05.176430 1514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50207
I20260620 02:52:05.177767 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 1585
I20260620 02:52:05.178007 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:52:05.178746 388 external_mini_cluster.cc:1428] Setting key 8cd0490082e73d029617ed38593a1601
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:52:05.219053 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38391
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:05.553296 1725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:05.553591 1725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:05.553752 1725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:05.561028 1725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:05.561128 1725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:05.561266 1725 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:52:05.561375 1725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:52:05.571673 1725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:05.573127 1725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:05.574440 1725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:05.587396 1730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:05.587455 1733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:05.587905 1731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:05.589119 1725 server_base.cc:1034] running on GCE node
I20260620 02:52:05.589695 1725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:05.590693 1725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:05.591933 1725 hybrid_clock.cc:648] HybridClock initialized: now 1781923925591849 us; error 63 us; skew 500 ppm
I20260620 02:52:05.592262 1725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:05 dist-test-slave-8p8c krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923925, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:05.595487 1725 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:52:05.596891 1725 webserver.cc:460] Webserver started at http://127.0.97.2:34703/ using document root <none> and password file <none>
I20260620 02:52:05.597690 1725 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:05.597776 1725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:05.598122 1725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:05.600971 1725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-1/data/instance:
uuid: "271d193fc4314739b335db51778d5ffc"
format_stamp: "Formatted at 2026-06-20 02:52:05 on dist-test-slave-8p8c"
server_key: "1f9b25120b47f6de38c8554d95fb617c"
server_key_iv: "1eadfa9fc1a25d8ada0709fe64d39dfa"
server_key_version: "encryptionkey@0"
I20260620 02:52:05.601768 1725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-1/wal/instance:
uuid: "271d193fc4314739b335db51778d5ffc"
format_stamp: "Formatted at 2026-06-20 02:52:05 on dist-test-slave-8p8c"
server_key: "1f9b25120b47f6de38c8554d95fb617c"
server_key_iv: "1eadfa9fc1a25d8ada0709fe64d39dfa"
server_key_version: "encryptionkey@0"
I20260620 02:52:05.606876 1725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 02:52:05.610755 1740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:05.612102 1725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:52:05.612229 1725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-1/wal
uuid: "271d193fc4314739b335db51778d5ffc"
format_stamp: "Formatted at 2026-06-20 02:52:05 on dist-test-slave-8p8c"
server_key: "1f9b25120b47f6de38c8554d95fb617c"
server_key_iv: "1eadfa9fc1a25d8ada0709fe64d39dfa"
server_key_version: "encryptionkey@0"
I20260620 02:52:05.612438 1725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:05.624902 1725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:05.630172 1725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:05.630537 1725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:05.633558 1725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:05.633663 1725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:05.633785 1725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:05.633850 1725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:52:05 dist-test-slave-8p8c krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923925, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:05.658844 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:05.634253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57233 (local address 127.0.97.62:38391)
0620 02:52:05.634456 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:05.634463 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:05.636135 (+ 1672us) server_negotiation.cc:407] Connection header received
0620 02:52:05.638335 (+ 2200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:05.638344 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:05.638512 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:05.638651 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:05.641058 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:05.642147 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:05.645488 (+ 3341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:05.645937 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:05.646054 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:05.651688 (+ 5634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:05.651695 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:05.651700 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:05.651769 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:05.654517 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:05.655613 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:05.655623 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:05.655629 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:05.655806 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:05.656470 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:05.656476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:05.656481 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:05.656866 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:05.656990 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:05.658301 (+ 1311us) server_negotiation.cc:299] Negotiation successful
0620 02:52:05.658573 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 02:52:05.661164 1749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:05.635153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38391 (local address 127.0.97.2:57233)
0620 02:52:05.635968 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:05.636015 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:05.637950 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:05.638846 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:05.638854 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:05.639729 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:05.640818 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:05.640852 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:05.642346 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:05.642352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:05.645241 (+ 2889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:05.645262 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:05.646217 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:05.646227 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:05.646651 (+ 424us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:05.647653 (+ 1002us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:05.647700 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:05.651433 (+ 3733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:05.654676 (+ 3243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:05.654686 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:05.654710 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:05.655439 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:05.655949 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:05.655954 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:05.655958 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:05.656300 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:05.657027 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:05.657109 (+ 82us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:05.657720 (+ 611us) client_negotiation.cc:769] Sending connection context
0620 02:52:05.658126 (+ 406us) client_negotiation.cc:240] Negotiation successful
0620 02:52:05.658875 (+ 749us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":386,"threads_started":1}
I20260620 02:52:05.687889 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:05.667851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52437 (local address 127.0.97.62:38391)
0620 02:52:05.668054 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:05.668063 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:05.668867 (+ 804us) server_negotiation.cc:407] Connection header received
0620 02:52:05.669120 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:05.669124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:05.669189 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:05.669274 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:05.671037 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:05.672222 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:05.674635 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:05.675313 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:05.675458 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:05.680331 (+ 4873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:05.680337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:05.680340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:05.680374 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:05.684083 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:05.685138 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:05.685147 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:05.685152 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:05.685330 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:05.685899 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:05.685907 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:05.685912 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:05.686301 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:05.686517 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:05.687502 (+ 985us) server_negotiation.cc:299] Negotiation successful
0620 02:52:05.687670 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:52:05.688174 1802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:05.667847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38391 (local address 127.0.97.2:52437)
0620 02:52:05.668730 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:05.668759 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:05.668984 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:05.669352 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:05.669360 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:05.670180 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:05.670766 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:05.670786 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:05.672370 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:05.672377 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:05.674425 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:05.674444 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:05.678047 (+ 3603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:05.678059 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:05.678314 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:05.679130 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:05.679158 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:05.680088 (+ 930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:05.684330 (+ 4242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:05.684342 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:05.684348 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:05.684967 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:05.685513 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:05.685521 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:05.685528 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:05.685732 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:05.686602 (+ 870us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:05.686612 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:05.686836 (+ 224us) client_negotiation.cc:769] Sending connection context
0620 02:52:05.687033 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 02:52:05.687505 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":414,"threads_started":1}
I20260620 02:52:05.700435 1725 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33681
I20260620 02:52:05.700462 1859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33681 every 8 connection(s)
I20260620 02:52:05.701741 1725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:52:05.704293 1860 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38391
I20260620 02:52:05.704548 1860 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:05.705260 1860 heartbeater.cc:511] Master 127.0.97.62:38391 requested a full tablet report, sending...
I20260620 02:52:05.706817 1514 ts_manager.cc:194] Registered new tserver with Master: 271d193fc4314739b335db51778d5ffc (127.0.97.2:33681)
I20260620 02:52:05.707698 1514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57233
I20260620 02:52:05.708142 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 1725
I20260620 02:52:05.708356 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:52:05.708832 388 external_mini_cluster.cc:1428] Setting key 35b10f38216ddcf412e27f67bfd14b56
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:52:05.755112 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38391
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:06.086766 1864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:06.087064 1864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:06.087244 1864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:06.094695 1864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:06.094806 1864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:06.094947 1864 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:52:06.095067 1864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:52:06.105310 1864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:06.106849 1864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:06.108213 1864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:06.121886 1870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:06.121868 1872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:06.121932 1869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:06.123502 1864 server_base.cc:1034] running on GCE node
I20260620 02:52:06.124109 1864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:06.125125 1864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:06.126479 1864 hybrid_clock.cc:648] HybridClock initialized: now 1781923926126316 us; error 46 us; skew 500 ppm
I20260620 02:52:06.126824 1864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:06 dist-test-slave-8p8c krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:06.130590 1864 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:52:06.132437 1864 webserver.cc:460] Webserver started at http://127.0.97.3:45383/ using document root <none> and password file <none>
I20260620 02:52:06.133203 1864 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:06.133347 1864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:06.133694 1864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:06.136905 1864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-2/data/instance:
uuid: "256f9de1d4de4561b4f186af956880ea"
format_stamp: "Formatted at 2026-06-20 02:52:06 on dist-test-slave-8p8c"
server_key: "28bf596af96d861ce03e11c31781677e"
server_key_iv: "2d3cbbacfa4461ba8bada8c831dd2856"
server_key_version: "encryptionkey@0"
I20260620 02:52:06.137717 1864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-2/wal/instance:
uuid: "256f9de1d4de4561b4f186af956880ea"
format_stamp: "Formatted at 2026-06-20 02:52:06 on dist-test-slave-8p8c"
server_key: "28bf596af96d861ce03e11c31781677e"
server_key_iv: "2d3cbbacfa4461ba8bada8c831dd2856"
server_key_version: "encryptionkey@0"
I20260620 02:52:06.143561 1864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 02:52:06.147295 1879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:06.148850 1864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:52:06.149055 1864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-2/wal
uuid: "256f9de1d4de4561b4f186af956880ea"
format_stamp: "Formatted at 2026-06-20 02:52:06 on dist-test-slave-8p8c"
server_key: "28bf596af96d861ce03e11c31781677e"
server_key_iv: "2d3cbbacfa4461ba8bada8c831dd2856"
server_key_version: "encryptionkey@0"
I20260620 02:52:06.149312 1864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:06.179608 1721 heartbeater.cc:503] Master 127.0.97.62:38391 was elected leader, sending a full tablet report...
I20260620 02:52:06.181774 1864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:06.186880 1864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:06.187237 1864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:06.190619 1864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:06.190752 1864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:06.190881 1864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:06.190951 1864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:52:06 dist-test-slave-8p8c krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:06.215749 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:06.191490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40369 (local address 127.0.97.62:38391)
0620 02:52:06.191993 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:06.191998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:06.193559 (+ 1561us) server_negotiation.cc:407] Connection header received
0620 02:52:06.195856 (+ 2297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:06.195861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:06.195951 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:06.196081 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:06.198523 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.199633 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.202172 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.202937 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.203111 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:06.207196 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:06.207203 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:06.207209 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:06.207276 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:06.211104 (+ 3828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:06.212526 (+ 1422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:06.212533 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:06.212537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:06.212687 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:06.213326 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:06.213329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:06.213331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:06.213639 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:06.213764 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:06.214962 (+ 1198us) server_negotiation.cc:299] Negotiation successful
0620 02:52:06.215234 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":253,"threads_started":1}
I20260620 02:52:06.217742 1889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:06.192232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38391 (local address 127.0.97.3:40369)
0620 02:52:06.193355 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:06.193395 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:06.195540 (+ 2145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:06.196392 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:06.196407 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:06.197280 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:06.198229 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:06.198261 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.199802 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.199807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:06.201996 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:06.202019 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.203243 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.203251 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:06.203529 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:06.204261 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:06.204283 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:06.206920 (+ 2637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:06.211381 (+ 4461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:06.211393 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:06.211412 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:06.212329 (+ 917us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:06.212819 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:06.212827 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:06.212833 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:06.213189 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:06.213774 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:06.213787 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:06.214428 (+ 641us) client_negotiation.cc:769] Sending connection context
0620 02:52:06.214827 (+ 399us) client_negotiation.cc:240] Negotiation successful
0620 02:52:06.215598 (+ 771us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"thread_start_us":463,"threads_started":1}
I20260620 02:52:06.245299 1864 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40121
I20260620 02:52:06.245319 1999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40121 every 8 connection(s)
I20260620 02:52:06.246996 1864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:52:06.249800 2000 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38391
I20260620 02:52:06.250051 2000 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:06.250880 2000 heartbeater.cc:511] Master 127.0.97.62:38391 requested a full tablet report, sending...
I20260620 02:52:06.252537 1514 ts_manager.cc:194] Registered new tserver with Master: 256f9de1d4de4561b4f186af956880ea (127.0.97.3:40121)
I20260620 02:52:06.253038 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:06.230589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54847 (local address 127.0.97.62:38391)
0620 02:52:06.230795 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:06.230803 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:06.232110 (+ 1307us) server_negotiation.cc:407] Connection header received
0620 02:52:06.232251 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:06.232260 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:06.232392 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:06.232545 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:06.233780 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.235090 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.237006 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.237441 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.237594 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:06.246458 (+ 8864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:06.246465 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:06.246469 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:06.246505 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:06.249987 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:06.250771 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:06.250776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:06.250780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:06.250942 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:06.251377 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:06.251382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:06.251385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:06.251734 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:06.251844 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:06.252380 (+ 536us) server_negotiation.cc:299] Negotiation successful
0620 02:52:06.252632 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 02:52:06.253072 1974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:06.231109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38391 (local address 127.0.97.3:54847)
0620 02:52:06.231948 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:06.231970 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:06.232133 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:06.232659 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:06.232662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:06.233211 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:06.233614 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:06.233627 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.235267 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.235271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:06.236830 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:06.236854 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.237634 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.237641 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:06.237858 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:06.244873 (+ 7015us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:06.244901 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:06.246116 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:06.250141 (+ 4025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:06.250149 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:06.250154 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:06.250651 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:06.251120 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:06.251124 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:06.251126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:06.251251 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:06.251852 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:06.251857 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:06.252097 (+ 240us) client_negotiation.cc:769] Sending connection context
0620 02:52:06.252274 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 02:52:06.252603 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":547,"threads_started":1}
I20260620 02:52:06.253659 1514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40369
I20260620 02:52:06.254206 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 1864
I20260620 02:52:06.254423 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:52:06.254830 388 external_mini_cluster.cc:1428] Setting key 02957340d347ac36ca143be93dab4d54
I20260620 02:52:06.258033 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 02:52:06 dist-test-slave-8p8c krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923926, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 02:52:06 dist-test-slave-8p8c krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923926, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:06.306419 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:06.291187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46226 (local address 127.0.97.62:38391)
0620 02:52:06.291349 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:06.291354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:06.291605 (+ 251us) server_negotiation.cc:407] Connection header received
0620 02:52:06.291752 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:06.291755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:06.291820 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:06.291912 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:06.293337 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.294141 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.295613 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.296010 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.296132 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:06.300335 (+ 4203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:06.300341 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:06.300344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:06.300383 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:06.303680 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:06.304361 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:06.304367 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:06.304401 (+ 34us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:06.304549 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:06.304989 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:06.304994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:06.304997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:06.305296 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:06.305506 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:06.305785 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 02:52:06.305956 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 02:52:06.311069 1513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46226:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:52:06.318529 1513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:52:06.353730 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:06.332111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33681 (local address 127.0.0.1:47426)
0620 02:52:06.343921 (+ 11810us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:06.344008 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:06.344266 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:06.345701 (+ 1435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:06.345711 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:06.345747 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:06.346410 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:06.346428 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.348541 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.348549 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:06.351287 (+ 2738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:06.351304 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.352582 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.352587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:06.352779 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:06.352908 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 02:52:06.353034 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 02:52:06.353227 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11432,"mutex_wait_us":2,"thread_start_us":12631,"threads_started":1}
I20260620 02:52:06.354223 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:06.332273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47426 (local address 127.0.97.2:33681)
0620 02:52:06.344815 (+ 12542us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:06.344824 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:06.344854 (+ 30us) server_negotiation.cc:407] Connection header received
0620 02:52:06.344948 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:06.344953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:06.345233 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:06.345440 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:06.346767 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.348410 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.351443 (+ 3033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.352319 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.352527 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:06.352767 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:06.353755 (+ 988us) server_negotiation.cc:299] Negotiation successful
0620 02:52:06.353961 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12386,"thread_start_us":12393,"threads_started":1}
I20260620 02:52:06.354502 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:06.332201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40297 (local address 127.0.0.1:48110)
0620 02:52:06.344444 (+ 12243us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:06.344469 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:06.344592 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:06.346530 (+ 1938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:06.346535 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:06.346555 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:06.346953 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:06.346967 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.348831 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.348837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:06.350991 (+ 2154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:06.351003 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.352932 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.352938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:06.353105 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:06.353132 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 02:52:06.353196 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 02:52:06.353328 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12104,"mutex_wait_us":8,"thread_start_us":11179,"threads_started":1}
I20260620 02:52:06.354961 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:06.332451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48110 (local address 127.0.97.1:40297)
0620 02:52:06.344829 (+ 12378us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:06.344838 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:06.344867 (+ 29us) server_negotiation.cc:407] Connection header received
0620 02:52:06.344965 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:06.344974 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:06.345233 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:06.345475 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:06.347143 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.348694 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.351154 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.352658 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.352781 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:06.354345 (+ 1564us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:06.354456 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 02:52:06.354677 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12214,"thread_start_us":12078,"threads_started":1}
I20260620 02:52:06.356796 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:06.345585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52906 (local address 127.0.97.3:40121)
0620 02:52:06.347740 (+ 2155us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:06.347748 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:06.347780 (+ 32us) server_negotiation.cc:407] Connection header received
0620 02:52:06.347868 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:06.347876 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:06.348142 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:06.348388 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:06.349526 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.351232 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.353618 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.355778 (+ 2160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.355949 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:06.356182 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:06.356329 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 02:52:06.356481 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1934,"thread_start_us":1756,"threads_started":1}
I20260620 02:52:06.356832 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:06.345190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40121 (local address 127.0.0.1:52906)
0620 02:52:06.346223 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:06.346239 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:06.346353 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:06.348608 (+ 2255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:06.348612 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:06.348642 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:06.349271 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:06.349283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.351373 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.351378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:06.353412 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:06.353429 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.355978 (+ 2549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.355985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:06.356160 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:06.356192 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 02:52:06.356289 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 02:52:06.356417 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"mutex_wait_us":27,"thread_start_us":710,"threads_started":1}
I20260620 02:52:06.358541 1651 tablet_service.cc:1467] Processing CreateTablet for tablet 091b554548b6420b92d5e38a0e29db43 (DEFAULT_TABLE table=test-table [id=76eb21300c9d48e8a6e0f4493e04a566]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:06.358539 1790 tablet_service.cc:1467] Processing CreateTablet for tablet 091b554548b6420b92d5e38a0e29db43 (DEFAULT_TABLE table=test-table [id=76eb21300c9d48e8a6e0f4493e04a566]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:06.360015 1651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 091b554548b6420b92d5e38a0e29db43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:06.360028 1790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 091b554548b6420b92d5e38a0e29db43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:06.361632 1930 tablet_service.cc:1467] Processing CreateTablet for tablet 091b554548b6420b92d5e38a0e29db43 (DEFAULT_TABLE table=test-table [id=76eb21300c9d48e8a6e0f4493e04a566]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:06.362989 1930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 091b554548b6420b92d5e38a0e29db43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:06.375279 2020 tablet_bootstrap.cc:492] T 091b554548b6420b92d5e38a0e29db43 P 173ed386c375473890ba5f27ebe29d22: Bootstrap starting.
I20260620 02:52:06.376266 2019 tablet_bootstrap.cc:492] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc: Bootstrap starting.
I20260620 02:52:06.377276 2021 tablet_bootstrap.cc:492] T 091b554548b6420b92d5e38a0e29db43 P 256f9de1d4de4561b4f186af956880ea: Bootstrap starting.
I20260620 02:52:06.380229 2020 tablet_bootstrap.cc:654] T 091b554548b6420b92d5e38a0e29db43 P 173ed386c375473890ba5f27ebe29d22: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:06.380503 2021 tablet_bootstrap.cc:654] T 091b554548b6420b92d5e38a0e29db43 P 256f9de1d4de4561b4f186af956880ea: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:06.381347 2020 log.cc:824] T 091b554548b6420b92d5e38a0e29db43 P 173ed386c375473890ba5f27ebe29d22: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:06.381466 2021 log.cc:824] T 091b554548b6420b92d5e38a0e29db43 P 256f9de1d4de4561b4f186af956880ea: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:06.384891 2019 tablet_bootstrap.cc:654] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:06.385105 2021 tablet_bootstrap.cc:492] T 091b554548b6420b92d5e38a0e29db43 P 256f9de1d4de4561b4f186af956880ea: No bootstrap required, opened a new log
I20260620 02:52:06.385135 2020 tablet_bootstrap.cc:492] T 091b554548b6420b92d5e38a0e29db43 P 173ed386c375473890ba5f27ebe29d22: No bootstrap required, opened a new log
I20260620 02:52:06.385388 2021 ts_tablet_manager.cc:1397] T 091b554548b6420b92d5e38a0e29db43 P 256f9de1d4de4561b4f186af956880ea: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 02:52:06.385388 2020 ts_tablet_manager.cc:1397] T 091b554548b6420b92d5e38a0e29db43 P 173ed386c375473890ba5f27ebe29d22: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260620 02:52:06.385982 2019 log.cc:824] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:06.389289 2019 tablet_bootstrap.cc:492] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc: No bootstrap required, opened a new log
I20260620 02:52:06.389577 2019 ts_tablet_manager.cc:1397] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.004s
I20260620 02:52:06.396358 2021 raft_consensus.cc:348] T 091b554548b6420b92d5e38a0e29db43 P 256f9de1d4de4561b4f186af956880ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } }
I20260620 02:52:06.396410 2020 raft_consensus.cc:348] T 091b554548b6420b92d5e38a0e29db43 P 173ed386c375473890ba5f27ebe29d22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } }
I20260620 02:52:06.396871 2021 raft_consensus.cc:374] T 091b554548b6420b92d5e38a0e29db43 P 256f9de1d4de4561b4f186af956880ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:06.396871 2020 raft_consensus.cc:374] T 091b554548b6420b92d5e38a0e29db43 P 173ed386c375473890ba5f27ebe29d22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:06.396986 2020 raft_consensus.cc:729] T 091b554548b6420b92d5e38a0e29db43 P 173ed386c375473890ba5f27ebe29d22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 173ed386c375473890ba5f27ebe29d22, State: Initialized, Role: FOLLOWER
I20260620 02:52:06.397003 2021 raft_consensus.cc:729] T 091b554548b6420b92d5e38a0e29db43 P 256f9de1d4de4561b4f186af956880ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 256f9de1d4de4561b4f186af956880ea, State: Initialized, Role: FOLLOWER
I20260620 02:52:06.396723 2019 raft_consensus.cc:348] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } }
I20260620 02:52:06.397184 2019 raft_consensus.cc:374] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:06.397300 2019 raft_consensus.cc:729] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 271d193fc4314739b335db51778d5ffc, State: Initialized, Role: FOLLOWER
I20260620 02:52:06.397706 2021 consensus_queue.cc:260] T 091b554548b6420b92d5e38a0e29db43 P 256f9de1d4de4561b4f186af956880ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } }
I20260620 02:52:06.397706 2020 consensus_queue.cc:260] T 091b554548b6420b92d5e38a0e29db43 P 173ed386c375473890ba5f27ebe29d22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } }
I20260620 02:52:06.397974 2019 consensus_queue.cc:260] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } }
I20260620 02:52:06.399626 2020 ts_tablet_manager.cc:1428] T 091b554548b6420b92d5e38a0e29db43 P 173ed386c375473890ba5f27ebe29d22: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260620 02:52:06.399626 2021 ts_tablet_manager.cc:1428] T 091b554548b6420b92d5e38a0e29db43 P 256f9de1d4de4561b4f186af956880ea: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
I20260620 02:52:06.400077 2000 heartbeater.cc:503] Master 127.0.97.62:38391 was elected leader, sending a full tablet report...
I20260620 02:52:06.401916 1860 heartbeater.cc:503] Master 127.0.97.62:38391 was elected leader, sending a full tablet report...
I20260620 02:52:06.403290 2019 ts_tablet_manager.cc:1428] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
W20260620 02:52:06.421702 1722 tablet.cc:2369] T 091b554548b6420b92d5e38a0e29db43 P 173ed386c375473890ba5f27ebe29d22: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:52:06.453830 1861 tablet.cc:2369] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:52:06.498886 2001 tablet.cc:2369] T 091b554548b6420b92d5e38a0e29db43 P 256f9de1d4de4561b4f186af956880ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:52:06.521927 2027 raft_consensus.cc:482] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:52:06.522401 2027 raft_consensus.cc:504] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } }
I20260620 02:52:06.525121 2027 leader_election.cc:288] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 173ed386c375473890ba5f27ebe29d22 (127.0.97.1:40297), 256f9de1d4de4561b4f186af956880ea (127.0.97.3:40121)
I20260620 02:52:06.536449 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:06.526092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49663 (local address 127.0.97.3:40121)
0620 02:52:06.526302 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:06.526311 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:06.526931 (+ 620us) server_negotiation.cc:407] Connection header received
0620 02:52:06.527371 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:06.527382 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:06.527491 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:06.527577 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:06.528955 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.530090 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.532616 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.534576 (+ 1960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.534759 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:06.534960 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:06.535817 (+ 857us) server_negotiation.cc:299] Negotiation successful
0620 02:52:06.536039 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:52:06.537599 2029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:06.525295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40121 (local address 127.0.97.2:49663)
0620 02:52:06.526773 (+ 1478us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:06.526800 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:06.527087 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:06.527893 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:06.527903 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:06.527947 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:06.528817 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:06.528828 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.530263 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.530268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:06.532400 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:06.532414 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.535177 (+ 2763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.535188 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:06.535539 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:06.535605 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 02:52:06.536792 (+ 1187us) client_negotiation.cc:240] Negotiation successful
0620 02:52:06.537023 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1252,"mutex_wait_us":338,"thread_start_us":1497,"threads_started":1}
I20260620 02:52:06.538846 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:06.526055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54553 (local address 127.0.97.1:40297)
0620 02:52:06.526262 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:06.526269 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:06.527228 (+ 959us) server_negotiation.cc:407] Connection header received
0620 02:52:06.527456 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:06.527474 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:06.527991 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:06.528058 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:06.532614 (+ 4556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.533664 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.536115 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.537323 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.537437 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:06.537549 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:06.538483 (+ 934us) server_negotiation.cc:299] Negotiation successful
0620 02:52:06.538625 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 02:52:06.538950 1950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "091b554548b6420b92d5e38a0e29db43" candidate_uuid: "271d193fc4314739b335db51778d5ffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "256f9de1d4de4561b4f186af956880ea" is_pre_election: true
I20260620 02:52:06.539600 2028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:06.525364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40297 (local address 127.0.97.2:54553)
0620 02:52:06.527131 (+ 1767us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:06.527149 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:06.527322 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:06.528235 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:06.528241 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:06.528268 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:06.528673 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:06.528685 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.533840 (+ 5155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.533845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:06.535933 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:06.535947 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.538089 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.538095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:06.538264 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:06.538295 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 02:52:06.539098 (+ 803us) client_negotiation.cc:240] Negotiation successful
0620 02:52:06.539247 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1639,"mutex_wait_us":43,"thread_start_us":974,"threads_started":1}
I20260620 02:52:06.539592 1950 raft_consensus.cc:2454] T 091b554548b6420b92d5e38a0e29db43 P 256f9de1d4de4561b4f186af956880ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 271d193fc4314739b335db51778d5ffc in term 0.
I20260620 02:52:06.540501 1742 leader_election.cc:302] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 256f9de1d4de4561b4f186af956880ea, 271d193fc4314739b335db51778d5ffc; no voters:
I20260620 02:52:06.540571 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "091b554548b6420b92d5e38a0e29db43" candidate_uuid: "271d193fc4314739b335db51778d5ffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "173ed386c375473890ba5f27ebe29d22" is_pre_election: true
I20260620 02:52:06.541074 2027 raft_consensus.cc:2781] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:52:06.541126 1671 raft_consensus.cc:2454] T 091b554548b6420b92d5e38a0e29db43 P 173ed386c375473890ba5f27ebe29d22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 271d193fc4314739b335db51778d5ffc in term 0.
I20260620 02:52:06.541239 2027 raft_consensus.cc:482] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:52:06.541355 2027 raft_consensus.cc:3037] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:06.545794 2027 raft_consensus.cc:504] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } }
I20260620 02:52:06.547096 2027 leader_election.cc:288] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc [CANDIDATE]: Term 1 election: Requested vote from peers 173ed386c375473890ba5f27ebe29d22 (127.0.97.1:40297), 256f9de1d4de4561b4f186af956880ea (127.0.97.3:40121)
I20260620 02:52:06.547363 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "091b554548b6420b92d5e38a0e29db43" candidate_uuid: "271d193fc4314739b335db51778d5ffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "173ed386c375473890ba5f27ebe29d22"
I20260620 02:52:06.547360 1950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "091b554548b6420b92d5e38a0e29db43" candidate_uuid: "271d193fc4314739b335db51778d5ffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "256f9de1d4de4561b4f186af956880ea"
I20260620 02:52:06.547601 1671 raft_consensus.cc:3037] T 091b554548b6420b92d5e38a0e29db43 P 173ed386c375473890ba5f27ebe29d22 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:06.547593 1950 raft_consensus.cc:3037] T 091b554548b6420b92d5e38a0e29db43 P 256f9de1d4de4561b4f186af956880ea [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:06.551050 1671 raft_consensus.cc:2454] T 091b554548b6420b92d5e38a0e29db43 P 173ed386c375473890ba5f27ebe29d22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 271d193fc4314739b335db51778d5ffc in term 1.
I20260620 02:52:06.551062 1950 raft_consensus.cc:2454] T 091b554548b6420b92d5e38a0e29db43 P 256f9de1d4de4561b4f186af956880ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 271d193fc4314739b335db51778d5ffc in term 1.
I20260620 02:52:06.551524 1742 leader_election.cc:302] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 256f9de1d4de4561b4f186af956880ea, 271d193fc4314739b335db51778d5ffc; no voters:
I20260620 02:52:06.551940 2027 raft_consensus.cc:2781] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:06.552558 2027 raft_consensus.cc:686] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc [term 1 LEADER]: Becoming Leader. State: Replica: 271d193fc4314739b335db51778d5ffc, State: Running, Role: LEADER
I20260620 02:52:06.553133 2027 consensus_queue.cc:237] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } }
I20260620 02:52:06.559345 1513 catalog_manager.cc:5495] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc reported cstate change: term changed from 0 to 1, leader changed from <none> to 271d193fc4314739b335db51778d5ffc (127.0.97.2). New cstate: current_term: 1 leader_uuid: "271d193fc4314739b335db51778d5ffc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } health_report { overall_health: UNKNOWN } } }
I20260620 02:52:06.611002 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:06.603835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47428 (local address 127.0.97.2:33681)
0620 02:52:06.604016 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:06.604040 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:06.604178 (+ 138us) server_negotiation.cc:407] Connection header received
0620 02:52:06.604597 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:06.604601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:06.604682 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:06.604809 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:52:06.605358 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.606079 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.608482 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.608911 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.609064 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:06.609740 (+ 676us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:52:06.610295 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:52:06.610494 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:06.610622 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 02:52:06.610729 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 02:52:06.621531 1950 raft_consensus.cc:1261] T 091b554548b6420b92d5e38a0e29db43 P 256f9de1d4de4561b4f186af956880ea [term 1 FOLLOWER]: Refusing update from remote peer 271d193fc4314739b335db51778d5ffc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:06.621531 1671 raft_consensus.cc:1261] T 091b554548b6420b92d5e38a0e29db43 P 173ed386c375473890ba5f27ebe29d22 [term 1 FOLLOWER]: Refusing update from remote peer 271d193fc4314739b335db51778d5ffc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:06.622633 2027 consensus_queue.cc:1035] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc [LEADER]: Connected to new peer: Peer: permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:52:06.623059 2030 consensus_queue.cc:1035] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc [LEADER]: Connected to new peer: Peer: permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:52:06.674757 1512 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:52:06.675104 1512 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:52:06.680084 1512 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32de122f2423469682096ed5bb608fa8: Sending DeleteTablet for 3 replicas of tablet 091b554548b6420b92d5e38a0e29db43
I20260620 02:52:06.681298 1651 tablet_service.cc:1514] Processing DeleteTablet for tablet 091b554548b6420b92d5e38a0e29db43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48110
I20260620 02:52:06.681362 1790 tablet_service.cc:1514] Processing DeleteTablet for tablet 091b554548b6420b92d5e38a0e29db43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47426
I20260620 02:52:06.682839 1930 tablet_service.cc:1514] Processing DeleteTablet for tablet 091b554548b6420b92d5e38a0e29db43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52906
I20260620 02:52:06.687836 1512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46226:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:52:06.688614 1512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:52:06.697821 2042 tablet_replica.cc:330] T 091b554548b6420b92d5e38a0e29db43 P 173ed386c375473890ba5f27ebe29d22: stopping tablet replica
I20260620 02:52:06.698508 2042 raft_consensus.cc:2229] T 091b554548b6420b92d5e38a0e29db43 P 173ed386c375473890ba5f27ebe29d22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:52:06.699115 2042 raft_consensus.cc:2258] T 091b554548b6420b92d5e38a0e29db43 P 173ed386c375473890ba5f27ebe29d22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:06.704392 1789 tablet_service.cc:1467] Processing CreateTablet for tablet a90d08e6f2ff4ba09e9a7ddc454585f0 (DEFAULT_TABLE table=test-table [id=904ddbe955764c37a1436cb9df5b5fbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:06.705116 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a90d08e6f2ff4ba09e9a7ddc454585f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:06.707163 2042 ts_tablet_manager.cc:1905] T 091b554548b6420b92d5e38a0e29db43 P 173ed386c375473890ba5f27ebe29d22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:06.707234 1929 tablet_service.cc:1467] Processing CreateTablet for tablet a90d08e6f2ff4ba09e9a7ddc454585f0 (DEFAULT_TABLE table=test-table [id=904ddbe955764c37a1436cb9df5b5fbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:06.707858 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a90d08e6f2ff4ba09e9a7ddc454585f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:06.715365 1651 tablet_service.cc:1467] Processing CreateTablet for tablet a90d08e6f2ff4ba09e9a7ddc454585f0 (DEFAULT_TABLE table=test-table [id=904ddbe955764c37a1436cb9df5b5fbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:06.718549 1651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a90d08e6f2ff4ba09e9a7ddc454585f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:06.718883 2044 tablet_replica.cc:330] T 091b554548b6420b92d5e38a0e29db43 P 256f9de1d4de4561b4f186af956880ea: stopping tablet replica
I20260620 02:52:06.719425 2044 raft_consensus.cc:2229] T 091b554548b6420b92d5e38a0e29db43 P 256f9de1d4de4561b4f186af956880ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:52:06.719928 2044 raft_consensus.cc:2258] T 091b554548b6420b92d5e38a0e29db43 P 256f9de1d4de4561b4f186af956880ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:06.723460 2042 ts_tablet_manager.cc:1918] T 091b554548b6420b92d5e38a0e29db43 P 173ed386c375473890ba5f27ebe29d22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:52:06.732524 2043 tablet_replica.cc:330] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc: stopping tablet replica
I20260620 02:52:06.733036 2043 raft_consensus.cc:2229] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:52:06.733750 2043 raft_consensus.cc:2258] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:06.734504 2042 log.cc:1196] T 091b554548b6420b92d5e38a0e29db43 P 173ed386c375473890ba5f27ebe29d22: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-0/wal/wals/091b554548b6420b92d5e38a0e29db43
I20260620 02:52:06.735155 2042 ts_tablet_manager.cc:1939] T 091b554548b6420b92d5e38a0e29db43 P 173ed386c375473890ba5f27ebe29d22: Deleting consensus metadata
I20260620 02:52:06.737069 1500 catalog_manager.cc:4841] TS 173ed386c375473890ba5f27ebe29d22 (127.0.97.1:40297): tablet 091b554548b6420b92d5e38a0e29db43 (table test-table [id=76eb21300c9d48e8a6e0f4493e04a566]) successfully deleted
I20260620 02:52:06.737267 2019 tablet_bootstrap.cc:492] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 271d193fc4314739b335db51778d5ffc: Bootstrap starting.
I20260620 02:52:06.737666 2021 tablet_bootstrap.cc:492] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 256f9de1d4de4561b4f186af956880ea: Bootstrap starting.
I20260620 02:52:06.739423 2019 tablet_bootstrap.cc:654] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 271d193fc4314739b335db51778d5ffc: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:06.739648 2021 tablet_bootstrap.cc:654] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 256f9de1d4de4561b4f186af956880ea: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:06.744223 2021 tablet_bootstrap.cc:492] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 256f9de1d4de4561b4f186af956880ea: No bootstrap required, opened a new log
I20260620 02:52:06.745471 2021 ts_tablet_manager.cc:1397] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 256f9de1d4de4561b4f186af956880ea: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260620 02:52:06.747076 2021 raft_consensus.cc:348] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 256f9de1d4de4561b4f186af956880ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } }
I20260620 02:52:06.747341 2021 raft_consensus.cc:374] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 256f9de1d4de4561b4f186af956880ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:06.747437 2021 raft_consensus.cc:729] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 256f9de1d4de4561b4f186af956880ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 256f9de1d4de4561b4f186af956880ea, State: Initialized, Role: FOLLOWER
I20260620 02:52:06.747728 2021 consensus_queue.cc:260] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 256f9de1d4de4561b4f186af956880ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } }
I20260620 02:52:06.748091 2043 ts_tablet_manager.cc:1905] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:06.748554 2021 ts_tablet_manager.cc:1428] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 256f9de1d4de4561b4f186af956880ea: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 02:52:06.749781 2019 tablet_bootstrap.cc:492] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 271d193fc4314739b335db51778d5ffc: No bootstrap required, opened a new log
I20260620 02:52:06.749996 2019 ts_tablet_manager.cc:1397] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 271d193fc4314739b335db51778d5ffc: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20260620 02:52:06.751520 2019 raft_consensus.cc:348] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 271d193fc4314739b335db51778d5ffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } }
I20260620 02:52:06.751789 2019 raft_consensus.cc:374] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 271d193fc4314739b335db51778d5ffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:06.751916 2019 raft_consensus.cc:729] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 271d193fc4314739b335db51778d5ffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 271d193fc4314739b335db51778d5ffc, State: Initialized, Role: FOLLOWER
I20260620 02:52:06.752269 2019 consensus_queue.cc:260] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 271d193fc4314739b335db51778d5ffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } }
I20260620 02:52:06.752867 2019 ts_tablet_manager.cc:1428] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 271d193fc4314739b335db51778d5ffc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 02:52:06.756261 2044 ts_tablet_manager.cc:1905] T 091b554548b6420b92d5e38a0e29db43 P 256f9de1d4de4561b4f186af956880ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:06.756532 2043 ts_tablet_manager.cc:1918] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:52:06.756722 2043 log.cc:1196] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-1/wal/wals/091b554548b6420b92d5e38a0e29db43
I20260620 02:52:06.757619 2043 ts_tablet_manager.cc:1939] T 091b554548b6420b92d5e38a0e29db43 P 271d193fc4314739b335db51778d5ffc: Deleting consensus metadata
I20260620 02:52:06.758008 2020 tablet_bootstrap.cc:492] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 173ed386c375473890ba5f27ebe29d22: Bootstrap starting.
I20260620 02:52:06.760098 2020 tablet_bootstrap.cc:654] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 173ed386c375473890ba5f27ebe29d22: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:06.761348 2026 raft_consensus.cc:482] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 256f9de1d4de4561b4f186af956880ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:52:06.761561 2026 raft_consensus.cc:504] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 256f9de1d4de4561b4f186af956880ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } }
I20260620 02:52:06.763741 2026 leader_election.cc:288] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 256f9de1d4de4561b4f186af956880ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 173ed386c375473890ba5f27ebe29d22 (127.0.97.1:40297), 271d193fc4314739b335db51778d5ffc (127.0.97.2:33681)
I20260620 02:52:06.764865 2044 ts_tablet_manager.cc:1918] T 091b554548b6420b92d5e38a0e29db43 P 256f9de1d4de4561b4f186af956880ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:52:06.765056 2044 log.cc:1196] T 091b554548b6420b92d5e38a0e29db43 P 256f9de1d4de4561b4f186af956880ea: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-2/wal/wals/091b554548b6420b92d5e38a0e29db43
I20260620 02:52:06.765607 2044 ts_tablet_manager.cc:1939] T 091b554548b6420b92d5e38a0e29db43 P 256f9de1d4de4561b4f186af956880ea: Deleting consensus metadata
I20260620 02:52:06.768576 1499 catalog_manager.cc:4841] TS 256f9de1d4de4561b4f186af956880ea (127.0.97.3:40121): tablet 091b554548b6420b92d5e38a0e29db43 (table test-table [id=76eb21300c9d48e8a6e0f4493e04a566]) successfully deleted
I20260620 02:52:06.769155 1499 catalog_manager.cc:4841] TS 271d193fc4314739b335db51778d5ffc (127.0.97.2:33681): tablet 091b554548b6420b92d5e38a0e29db43 (table test-table [id=76eb21300c9d48e8a6e0f4493e04a566]) successfully deleted
I20260620 02:52:06.780211 2020 tablet_bootstrap.cc:492] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 173ed386c375473890ba5f27ebe29d22: No bootstrap required, opened a new log
I20260620 02:52:06.780474 2020 ts_tablet_manager.cc:1397] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 173ed386c375473890ba5f27ebe29d22: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.005s
I20260620 02:52:06.781495 2020 raft_consensus.cc:348] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 173ed386c375473890ba5f27ebe29d22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } }
I20260620 02:52:06.781858 2020 raft_consensus.cc:374] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 173ed386c375473890ba5f27ebe29d22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:06.781999 2020 raft_consensus.cc:729] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 173ed386c375473890ba5f27ebe29d22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 173ed386c375473890ba5f27ebe29d22, State: Initialized, Role: FOLLOWER
I20260620 02:52:06.782451 2020 consensus_queue.cc:260] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 173ed386c375473890ba5f27ebe29d22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } }
I20260620 02:52:06.784947 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:06.766165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41243 (local address 127.0.97.2:33681)
0620 02:52:06.766391 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:06.766400 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:06.775235 (+ 8835us) server_negotiation.cc:407] Connection header received
0620 02:52:06.775405 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:06.775413 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:06.775503 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:06.775652 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:06.779063 (+ 3411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.779993 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.782417 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.783720 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.783840 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:06.783951 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:06.784495 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 02:52:06.784628 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 02:52:06.785687 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:06.765783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33681 (local address 127.0.97.3:41243)
0620 02:52:06.775141 (+ 9358us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:06.775159 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:06.775279 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:06.777555 (+ 2276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:06.777561 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:06.777591 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:06.777986 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:06.777999 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.780137 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.780145 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:06.782200 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:06.782217 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.784089 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.784095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:06.784295 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:06.784334 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 02:52:06.785177 (+ 843us) client_negotiation.cc:240] Negotiation successful
0620 02:52:06.785324 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9224,"thread_start_us":8064,"threads_started":1}
I20260620 02:52:06.786285 2020 ts_tablet_manager.cc:1428] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 173ed386c375473890ba5f27ebe29d22: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260620 02:52:06.786648 1815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a90d08e6f2ff4ba09e9a7ddc454585f0" candidate_uuid: "256f9de1d4de4561b4f186af956880ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "271d193fc4314739b335db51778d5ffc" is_pre_election: true
I20260620 02:52:06.787104 1815 raft_consensus.cc:2454] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 271d193fc4314739b335db51778d5ffc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 256f9de1d4de4561b4f186af956880ea in term 0.
I20260620 02:52:06.790093 1881 leader_election.cc:302] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 256f9de1d4de4561b4f186af956880ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 256f9de1d4de4561b4f186af956880ea, 271d193fc4314739b335db51778d5ffc; no voters:
I20260620 02:52:06.790599 2026 raft_consensus.cc:2781] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 256f9de1d4de4561b4f186af956880ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:52:06.790735 2026 raft_consensus.cc:482] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 256f9de1d4de4561b4f186af956880ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:52:06.791172 2026 raft_consensus.cc:3037] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 256f9de1d4de4561b4f186af956880ea [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:06.792004 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:06.764948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59411 (local address 127.0.97.1:40297)
0620 02:52:06.765152 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:06.765160 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:06.774258 (+ 9098us) server_negotiation.cc:407] Connection header received
0620 02:52:06.775860 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:06.775868 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:06.775993 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:06.776097 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:06.778182 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.780053 (+ 1871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.788931 (+ 8878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.790277 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.790396 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:06.790510 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:06.791594 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0620 02:52:06.791737 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 02:52:06.792661 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:06.764467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40297 (local address 127.0.97.3:59411)
0620 02:52:06.774142 (+ 9675us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:06.774167 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:06.774319 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:06.776868 (+ 2549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:06.776875 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:06.776909 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:06.777279 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:06.777293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.786605 (+ 9312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.786612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:06.788741 (+ 2129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:06.788756 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.791182 (+ 2426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.791191 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:06.791382 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:06.791419 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 02:52:06.792149 (+ 730us) client_negotiation.cc:240] Negotiation successful
0620 02:52:06.792303 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9538,"thread_start_us":10354,"threads_started":1}
I20260620 02:52:06.793458 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a90d08e6f2ff4ba09e9a7ddc454585f0" candidate_uuid: "256f9de1d4de4561b4f186af956880ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "173ed386c375473890ba5f27ebe29d22" is_pre_election: true
I20260620 02:52:06.793757 1670 raft_consensus.cc:2454] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 173ed386c375473890ba5f27ebe29d22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 256f9de1d4de4561b4f186af956880ea in term 0.
I20260620 02:52:06.795295 2026 raft_consensus.cc:504] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 256f9de1d4de4561b4f186af956880ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } }
I20260620 02:52:06.796645 2026 leader_election.cc:288] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 256f9de1d4de4561b4f186af956880ea [CANDIDATE]: Term 1 election: Requested vote from peers 173ed386c375473890ba5f27ebe29d22 (127.0.97.1:40297), 271d193fc4314739b335db51778d5ffc (127.0.97.2:33681)
I20260620 02:52:06.796765 1815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a90d08e6f2ff4ba09e9a7ddc454585f0" candidate_uuid: "256f9de1d4de4561b4f186af956880ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "271d193fc4314739b335db51778d5ffc"
I20260620 02:52:06.797003 1815 raft_consensus.cc:3037] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 271d193fc4314739b335db51778d5ffc [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:06.798595 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a90d08e6f2ff4ba09e9a7ddc454585f0" candidate_uuid: "256f9de1d4de4561b4f186af956880ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "173ed386c375473890ba5f27ebe29d22"
I20260620 02:52:06.798802 1670 raft_consensus.cc:3037] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 173ed386c375473890ba5f27ebe29d22 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:06.801738 1670 raft_consensus.cc:2454] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 173ed386c375473890ba5f27ebe29d22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 256f9de1d4de4561b4f186af956880ea in term 1.
I20260620 02:52:06.802189 1882 leader_election.cc:302] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 256f9de1d4de4561b4f186af956880ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 173ed386c375473890ba5f27ebe29d22, 256f9de1d4de4561b4f186af956880ea; no voters:
I20260620 02:52:06.802673 2026 raft_consensus.cc:2781] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 256f9de1d4de4561b4f186af956880ea [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:06.804363 1815 raft_consensus.cc:2454] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 271d193fc4314739b335db51778d5ffc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 256f9de1d4de4561b4f186af956880ea in term 1.
I20260620 02:52:06.805302 2026 raft_consensus.cc:686] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 256f9de1d4de4561b4f186af956880ea [term 1 LEADER]: Becoming Leader. State: Replica: 256f9de1d4de4561b4f186af956880ea, State: Running, Role: LEADER
I20260620 02:52:06.805783 2026 consensus_queue.cc:237] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 256f9de1d4de4561b4f186af956880ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } }
I20260620 02:52:06.810297 1512 catalog_manager.cc:5495] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 256f9de1d4de4561b4f186af956880ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 256f9de1d4de4561b4f186af956880ea (127.0.97.3). New cstate: current_term: 1 leader_uuid: "256f9de1d4de4561b4f186af956880ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } health_report { overall_health: HEALTHY } } }
Jun 20 02:52:06 dist-test-slave-8p8c krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:06.869261 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:06.853232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46240 (local address 127.0.97.62:38391)
0620 02:52:06.854113 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:06.854119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:06.854340 (+ 221us) server_negotiation.cc:407] Connection header received
0620 02:52:06.854599 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:06.854602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:06.854658 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:06.854773 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:06.856114 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.856879 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.858787 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.859238 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.859373 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:06.862897 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:06.862902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:06.862906 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:06.862933 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:06.865970 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:06.866744 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:06.866749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:06.866753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:06.867063 (+ 310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:06.867584 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:06.867594 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:06.867597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:06.867883 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:06.868075 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:06.868477 (+ 402us) server_negotiation.cc:299] Negotiation successful
0620 02:52:06.868732 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":470,"threads_started":1}
I20260620 02:52:06.869372 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:06.853325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38391 (local address 127.0.0.1:46240)
0620 02:52:06.854235 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:06.854251 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:06.854445 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:06.854793 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:06.854797 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:06.855761 (+ 964us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:06.855956 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:06.855966 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.857040 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.857052 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:06.858646 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:06.858660 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:06.859388 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:06.859394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:06.859571 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:06.860324 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:06.860343 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:06.862709 (+ 2366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:06.866091 (+ 3382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:06.866104 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:06.866113 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:06.866626 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:06.867216 (+ 590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:06.867221 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:06.867225 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:06.867367 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:06.868016 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:06.868024 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:06.868210 (+ 186us) client_negotiation.cc:769] Sending connection context
0620 02:52:06.868380 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 02:52:06.868901 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":129,"thread_start_us":618,"threads_started":1}
I20260620 02:52:06.875511 1512 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46240:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 02:52:06.876782 1512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:52:06.884079 1651 tablet_service.cc:1467] Processing CreateTablet for tablet 7df4f51c34bf49c2ad7c5a87f8854138 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c2f4856e0d9a435dbf2a1518ddee8217]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:52:06.884686 1651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7df4f51c34bf49c2ad7c5a87f8854138. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:06.884644 1789 tablet_service.cc:1467] Processing CreateTablet for tablet 7df4f51c34bf49c2ad7c5a87f8854138 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c2f4856e0d9a435dbf2a1518ddee8217]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:52:06.885145 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7df4f51c34bf49c2ad7c5a87f8854138. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:06.891686 2019 tablet_bootstrap.cc:492] T 7df4f51c34bf49c2ad7c5a87f8854138 P 271d193fc4314739b335db51778d5ffc: Bootstrap starting.
I20260620 02:52:06.892416 2020 tablet_bootstrap.cc:492] T 7df4f51c34bf49c2ad7c5a87f8854138 P 173ed386c375473890ba5f27ebe29d22: Bootstrap starting.
I20260620 02:52:06.893169 1929 tablet_service.cc:1467] Processing CreateTablet for tablet 7df4f51c34bf49c2ad7c5a87f8854138 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c2f4856e0d9a435dbf2a1518ddee8217]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:52:06.893649 2019 tablet_bootstrap.cc:654] T 7df4f51c34bf49c2ad7c5a87f8854138 P 271d193fc4314739b335db51778d5ffc: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:06.893675 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7df4f51c34bf49c2ad7c5a87f8854138. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:06.894419 2020 tablet_bootstrap.cc:654] T 7df4f51c34bf49c2ad7c5a87f8854138 P 173ed386c375473890ba5f27ebe29d22: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:06.901764 2021 tablet_bootstrap.cc:492] T 7df4f51c34bf49c2ad7c5a87f8854138 P 256f9de1d4de4561b4f186af956880ea: Bootstrap starting.
I20260620 02:52:06.903475 2021 tablet_bootstrap.cc:654] T 7df4f51c34bf49c2ad7c5a87f8854138 P 256f9de1d4de4561b4f186af956880ea: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:06.905640 2019 tablet_bootstrap.cc:492] T 7df4f51c34bf49c2ad7c5a87f8854138 P 271d193fc4314739b335db51778d5ffc: No bootstrap required, opened a new log
I20260620 02:52:06.905933 2019 ts_tablet_manager.cc:1397] T 7df4f51c34bf49c2ad7c5a87f8854138 P 271d193fc4314739b335db51778d5ffc: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20260620 02:52:06.906689 2020 tablet_bootstrap.cc:492] T 7df4f51c34bf49c2ad7c5a87f8854138 P 173ed386c375473890ba5f27ebe29d22: No bootstrap required, opened a new log
I20260620 02:52:06.906934 2020 ts_tablet_manager.cc:1397] T 7df4f51c34bf49c2ad7c5a87f8854138 P 173ed386c375473890ba5f27ebe29d22: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.010s
I20260620 02:52:06.906960 2019 raft_consensus.cc:348] T 7df4f51c34bf49c2ad7c5a87f8854138 P 271d193fc4314739b335db51778d5ffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } }
I20260620 02:52:06.907320 2019 raft_consensus.cc:374] T 7df4f51c34bf49c2ad7c5a87f8854138 P 271d193fc4314739b335db51778d5ffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:06.907454 2019 raft_consensus.cc:729] T 7df4f51c34bf49c2ad7c5a87f8854138 P 271d193fc4314739b335db51778d5ffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 271d193fc4314739b335db51778d5ffc, State: Initialized, Role: FOLLOWER
I20260620 02:52:06.907819 2020 raft_consensus.cc:348] T 7df4f51c34bf49c2ad7c5a87f8854138 P 173ed386c375473890ba5f27ebe29d22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } }
I20260620 02:52:06.907855 2019 consensus_queue.cc:260] T 7df4f51c34bf49c2ad7c5a87f8854138 P 271d193fc4314739b335db51778d5ffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } }
I20260620 02:52:06.908114 2020 raft_consensus.cc:374] T 7df4f51c34bf49c2ad7c5a87f8854138 P 173ed386c375473890ba5f27ebe29d22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:06.908205 2020 raft_consensus.cc:729] T 7df4f51c34bf49c2ad7c5a87f8854138 P 173ed386c375473890ba5f27ebe29d22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 173ed386c375473890ba5f27ebe29d22, State: Initialized, Role: FOLLOWER
I20260620 02:52:06.908455 2020 consensus_queue.cc:260] T 7df4f51c34bf49c2ad7c5a87f8854138 P 173ed386c375473890ba5f27ebe29d22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } }
I20260620 02:52:06.908560 2027 tablet_replica.cc:439] T 7df4f51c34bf49c2ad7c5a87f8854138 P 271d193fc4314739b335db51778d5ffc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } } }
I20260620 02:52:06.908783 2019 ts_tablet_manager.cc:1428] T 7df4f51c34bf49c2ad7c5a87f8854138 P 271d193fc4314739b335db51778d5ffc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 02:52:06.908988 2027 tablet_replica.cc:442] T 7df4f51c34bf49c2ad7c5a87f8854138 P 271d193fc4314739b335db51778d5ffc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:52:06.908962 2025 tablet_replica.cc:439] T 7df4f51c34bf49c2ad7c5a87f8854138 P 173ed386c375473890ba5f27ebe29d22: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } } }
I20260620 02:52:06.909233 2025 tablet_replica.cc:442] T 7df4f51c34bf49c2ad7c5a87f8854138 P 173ed386c375473890ba5f27ebe29d22: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:52:06.910075 2020 ts_tablet_manager.cc:1428] T 7df4f51c34bf49c2ad7c5a87f8854138 P 173ed386c375473890ba5f27ebe29d22: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 02:52:06.916244 2021 tablet_bootstrap.cc:492] T 7df4f51c34bf49c2ad7c5a87f8854138 P 256f9de1d4de4561b4f186af956880ea: No bootstrap required, opened a new log
I20260620 02:52:06.916450 2021 ts_tablet_manager.cc:1397] T 7df4f51c34bf49c2ad7c5a87f8854138 P 256f9de1d4de4561b4f186af956880ea: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.008s
I20260620 02:52:06.917263 2021 raft_consensus.cc:348] T 7df4f51c34bf49c2ad7c5a87f8854138 P 256f9de1d4de4561b4f186af956880ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } }
I20260620 02:52:06.917526 2021 raft_consensus.cc:374] T 7df4f51c34bf49c2ad7c5a87f8854138 P 256f9de1d4de4561b4f186af956880ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:06.917598 2021 raft_consensus.cc:729] T 7df4f51c34bf49c2ad7c5a87f8854138 P 256f9de1d4de4561b4f186af956880ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 256f9de1d4de4561b4f186af956880ea, State: Initialized, Role: FOLLOWER
I20260620 02:52:06.917956 2021 consensus_queue.cc:260] T 7df4f51c34bf49c2ad7c5a87f8854138 P 256f9de1d4de4561b4f186af956880ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } }
I20260620 02:52:06.918684 2026 tablet_replica.cc:439] T 7df4f51c34bf49c2ad7c5a87f8854138 P 256f9de1d4de4561b4f186af956880ea: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } } }
I20260620 02:52:06.918886 2021 ts_tablet_manager.cc:1428] T 7df4f51c34bf49c2ad7c5a87f8854138 P 256f9de1d4de4561b4f186af956880ea: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 02:52:06.919009 2026 tablet_replica.cc:442] T 7df4f51c34bf49c2ad7c5a87f8854138 P 256f9de1d4de4561b4f186af956880ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:52:07.111860 2027 raft_consensus.cc:482] T 7df4f51c34bf49c2ad7c5a87f8854138 P 271d193fc4314739b335db51778d5ffc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:52:07.112087 2027 raft_consensus.cc:504] T 7df4f51c34bf49c2ad7c5a87f8854138 P 271d193fc4314739b335db51778d5ffc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } }
I20260620 02:52:07.112951 2027 leader_election.cc:288] T 7df4f51c34bf49c2ad7c5a87f8854138 P 271d193fc4314739b335db51778d5ffc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 173ed386c375473890ba5f27ebe29d22 (127.0.97.1:40297), 256f9de1d4de4561b4f186af956880ea (127.0.97.3:40121)
I20260620 02:52:07.113689 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df4f51c34bf49c2ad7c5a87f8854138" candidate_uuid: "271d193fc4314739b335db51778d5ffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "173ed386c375473890ba5f27ebe29d22" is_pre_election: true
I20260620 02:52:07.113740 1950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df4f51c34bf49c2ad7c5a87f8854138" candidate_uuid: "271d193fc4314739b335db51778d5ffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "256f9de1d4de4561b4f186af956880ea" is_pre_election: true
I20260620 02:52:07.113976 1670 raft_consensus.cc:2454] T 7df4f51c34bf49c2ad7c5a87f8854138 P 173ed386c375473890ba5f27ebe29d22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 271d193fc4314739b335db51778d5ffc in term 0.
I20260620 02:52:07.114025 1950 raft_consensus.cc:2454] T 7df4f51c34bf49c2ad7c5a87f8854138 P 256f9de1d4de4561b4f186af956880ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 271d193fc4314739b335db51778d5ffc in term 0.
I20260620 02:52:07.114542 1743 leader_election.cc:302] T 7df4f51c34bf49c2ad7c5a87f8854138 P 271d193fc4314739b335db51778d5ffc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 173ed386c375473890ba5f27ebe29d22, 271d193fc4314739b335db51778d5ffc; no voters:
I20260620 02:52:07.115020 2027 raft_consensus.cc:2781] T 7df4f51c34bf49c2ad7c5a87f8854138 P 271d193fc4314739b335db51778d5ffc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:52:07.115160 2027 raft_consensus.cc:482] T 7df4f51c34bf49c2ad7c5a87f8854138 P 271d193fc4314739b335db51778d5ffc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:52:07.115276 2027 raft_consensus.cc:3037] T 7df4f51c34bf49c2ad7c5a87f8854138 P 271d193fc4314739b335db51778d5ffc [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:07.119457 2027 raft_consensus.cc:504] T 7df4f51c34bf49c2ad7c5a87f8854138 P 271d193fc4314739b335db51778d5ffc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } }
I20260620 02:52:07.120440 2027 leader_election.cc:288] T 7df4f51c34bf49c2ad7c5a87f8854138 P 271d193fc4314739b335db51778d5ffc [CANDIDATE]: Term 1 election: Requested vote from peers 173ed386c375473890ba5f27ebe29d22 (127.0.97.1:40297), 256f9de1d4de4561b4f186af956880ea (127.0.97.3:40121)
I20260620 02:52:07.121066 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df4f51c34bf49c2ad7c5a87f8854138" candidate_uuid: "271d193fc4314739b335db51778d5ffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "173ed386c375473890ba5f27ebe29d22"
I20260620 02:52:07.121160 1950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df4f51c34bf49c2ad7c5a87f8854138" candidate_uuid: "271d193fc4314739b335db51778d5ffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "256f9de1d4de4561b4f186af956880ea"
I20260620 02:52:07.121325 1670 raft_consensus.cc:3037] T 7df4f51c34bf49c2ad7c5a87f8854138 P 173ed386c375473890ba5f27ebe29d22 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:07.121408 1950 raft_consensus.cc:3037] T 7df4f51c34bf49c2ad7c5a87f8854138 P 256f9de1d4de4561b4f186af956880ea [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:07.123525 1950 raft_consensus.cc:2454] T 7df4f51c34bf49c2ad7c5a87f8854138 P 256f9de1d4de4561b4f186af956880ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 271d193fc4314739b335db51778d5ffc in term 1.
I20260620 02:52:07.124018 1742 leader_election.cc:302] T 7df4f51c34bf49c2ad7c5a87f8854138 P 271d193fc4314739b335db51778d5ffc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 256f9de1d4de4561b4f186af956880ea, 271d193fc4314739b335db51778d5ffc; no voters:
I20260620 02:52:07.124313 2027 raft_consensus.cc:2781] T 7df4f51c34bf49c2ad7c5a87f8854138 P 271d193fc4314739b335db51778d5ffc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:07.124496 2027 raft_consensus.cc:686] T 7df4f51c34bf49c2ad7c5a87f8854138 P 271d193fc4314739b335db51778d5ffc [term 1 LEADER]: Becoming Leader. State: Replica: 271d193fc4314739b335db51778d5ffc, State: Running, Role: LEADER
I20260620 02:52:07.124888 1670 raft_consensus.cc:2454] T 7df4f51c34bf49c2ad7c5a87f8854138 P 173ed386c375473890ba5f27ebe29d22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 271d193fc4314739b335db51778d5ffc in term 1.
I20260620 02:52:07.124816 2027 consensus_queue.cc:237] T 7df4f51c34bf49c2ad7c5a87f8854138 P 271d193fc4314739b335db51778d5ffc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } }
I20260620 02:52:07.126959 2030 tablet_replica.cc:439] T 7df4f51c34bf49c2ad7c5a87f8854138 P 271d193fc4314739b335db51778d5ffc: TxnStatusTablet state changed. Reason: New leader 271d193fc4314739b335db51778d5ffc. Latest consensus state: current_term: 1 leader_uuid: "271d193fc4314739b335db51778d5ffc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } } }
I20260620 02:52:07.127334 2030 tablet_replica.cc:442] T 7df4f51c34bf49c2ad7c5a87f8854138 P 271d193fc4314739b335db51778d5ffc: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:52:07.128131 2063 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 02:52:07.128408 2063 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 02:52:07.129160 1512 catalog_manager.cc:5495] T 7df4f51c34bf49c2ad7c5a87f8854138 P 271d193fc4314739b335db51778d5ffc reported cstate change: term changed from 0 to 1, leader changed from <none> to 271d193fc4314739b335db51778d5ffc (127.0.97.2). New cstate: current_term: 1 leader_uuid: "271d193fc4314739b335db51778d5ffc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } health_report { overall_health: UNKNOWN } } }
I20260620 02:52:07.200237 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:07.194744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47434 (local address 127.0.97.2:33681)
0620 02:52:07.194978 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:07.194985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:07.195208 (+ 223us) server_negotiation.cc:407] Connection header received
0620 02:52:07.195521 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:07.195527 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:07.195618 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:07.195723 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:52:07.196083 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:07.196786 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:07.198511 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:07.198932 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:07.199077 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:07.199562 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:52:07.199659 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:52:07.199755 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:07.199886 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 02:52:07.199960 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 02:52:07.200291 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:07.194933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33681 (local address 127.0.0.1:47434)
0620 02:52:07.195125 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:07.195147 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:07.195296 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:07.195758 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:07.195762 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:07.195775 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:52:07.195955 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:07.195964 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:07.196921 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:07.196927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:07.198345 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:07.198358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:07.199094 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:07.199101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:07.199282 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:07.199334 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:52:07.199755 (+ 421us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:52:07.199776 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 02:52:07.199855 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 02:52:07.200007 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 02:52:07.204280 1950 raft_consensus.cc:1261] T 7df4f51c34bf49c2ad7c5a87f8854138 P 256f9de1d4de4561b4f186af956880ea [term 1 FOLLOWER]: Refusing update from remote peer 271d193fc4314739b335db51778d5ffc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:07.204411 1670 raft_consensus.cc:1261] T 7df4f51c34bf49c2ad7c5a87f8854138 P 173ed386c375473890ba5f27ebe29d22 [term 1 FOLLOWER]: Refusing update from remote peer 271d193fc4314739b335db51778d5ffc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:07.204843 2030 consensus_queue.cc:1035] T 7df4f51c34bf49c2ad7c5a87f8854138 P 271d193fc4314739b335db51778d5ffc [LEADER]: Connected to new peer: Peer: permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:52:07.205183 2027 consensus_queue.cc:1035] T 7df4f51c34bf49c2ad7c5a87f8854138 P 271d193fc4314739b335db51778d5ffc [LEADER]: Connected to new peer: Peer: permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:52:07.211227 2025 tablet_replica.cc:439] T 7df4f51c34bf49c2ad7c5a87f8854138 P 173ed386c375473890ba5f27ebe29d22: TxnStatusTablet state changed. Reason: New leader 271d193fc4314739b335db51778d5ffc. Latest consensus state: current_term: 1 leader_uuid: "271d193fc4314739b335db51778d5ffc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } } }
I20260620 02:52:07.211560 2025 tablet_replica.cc:442] T 7df4f51c34bf49c2ad7c5a87f8854138 P 173ed386c375473890ba5f27ebe29d22: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:52:07.217952 2065 mvcc.cc:205] Tried to move back new op lower bound from 7298760405822898176 to 7298760405506146304. Current Snapshot: MvccSnapshot[applied={T|T < 7298760405822898176}]
I20260620 02:52:07.223040 2025 tablet_replica.cc:439] T 7df4f51c34bf49c2ad7c5a87f8854138 P 173ed386c375473890ba5f27ebe29d22: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "271d193fc4314739b335db51778d5ffc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } } }
I20260620 02:52:07.223546 2025 tablet_replica.cc:442] T 7df4f51c34bf49c2ad7c5a87f8854138 P 173ed386c375473890ba5f27ebe29d22: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:52:07.226382 2026 tablet_replica.cc:439] T 7df4f51c34bf49c2ad7c5a87f8854138 P 256f9de1d4de4561b4f186af956880ea: TxnStatusTablet state changed. Reason: New leader 271d193fc4314739b335db51778d5ffc. Latest consensus state: current_term: 1 leader_uuid: "271d193fc4314739b335db51778d5ffc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } } }
I20260620 02:52:07.226693 2026 tablet_replica.cc:442] T 7df4f51c34bf49c2ad7c5a87f8854138 P 256f9de1d4de4561b4f186af956880ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:52:07.225387 2027 tablet_replica.cc:439] T 7df4f51c34bf49c2ad7c5a87f8854138 P 271d193fc4314739b335db51778d5ffc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "271d193fc4314739b335db51778d5ffc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } } }
I20260620 02:52:07.228968 2027 tablet_replica.cc:442] T 7df4f51c34bf49c2ad7c5a87f8854138 P 271d193fc4314739b335db51778d5ffc: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:52:07.230010 2027 tablet_replica.cc:439] T 7df4f51c34bf49c2ad7c5a87f8854138 P 271d193fc4314739b335db51778d5ffc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "271d193fc4314739b335db51778d5ffc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } } }
I20260620 02:52:07.230244 2027 tablet_replica.cc:442] T 7df4f51c34bf49c2ad7c5a87f8854138 P 271d193fc4314739b335db51778d5ffc: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:52:07.233525 2026 tablet_replica.cc:439] T 7df4f51c34bf49c2ad7c5a87f8854138 P 256f9de1d4de4561b4f186af956880ea: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "271d193fc4314739b335db51778d5ffc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 } } peers { permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 } } peers { permanent_uuid: "256f9de1d4de4561b4f186af956880ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40121 } } }
I20260620 02:52:07.233806 2026 tablet_replica.cc:442] T 7df4f51c34bf49c2ad7c5a87f8854138 P 256f9de1d4de4561b4f186af956880ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:52:07.246032 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:07.237479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52908 (local address 127.0.97.3:40121)
0620 02:52:07.238872 (+ 1393us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:07.238877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:07.238896 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:52:07.238973 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:07.238976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:07.239055 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:07.239194 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:52:07.239809 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:07.240990 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:07.243229 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:07.243802 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:07.243911 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:07.244520 (+ 609us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:52:07.245314 (+ 794us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:52:07.245468 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:07.245600 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 02:52:07.245678 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1271,"thread_start_us":1123,"threads_started":1}
I20260620 02:52:07.267036 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:07.258073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38055 (local address 127.0.97.2:33681)
0620 02:52:07.258268 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:07.258276 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:07.258964 (+ 688us) server_negotiation.cc:407] Connection header received
0620 02:52:07.259252 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:07.259258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:07.259340 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:07.259401 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:52:07.260174 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:07.261472 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:07.264098 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:07.264869 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:07.264973 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:07.265727 (+ 754us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:52:07.265903 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:52:07.265975 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:07.266251 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 02:52:07.266352 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 02:52:07.267973 2079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:07.258189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33681 (local address 127.0.97.3:38055)
0620 02:52:07.258845 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:07.258868 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:07.259089 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:07.259585 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:07.259591 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:07.259604 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:52:07.259975 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:07.259995 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:07.261672 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:07.261677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:07.263905 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:07.263922 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:07.265216 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:07.265224 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:07.265418 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:07.265496 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:52:07.266095 (+ 599us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:52:07.266106 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 02:52:07.267245 (+ 1139us) client_negotiation.cc:240] Negotiation successful
0620 02:52:07.267412 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":7873,"threads_started":1}
I20260620 02:52:07.283135 1670 raft_consensus.cc:1261] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 173ed386c375473890ba5f27ebe29d22 [term 1 FOLLOWER]: Refusing update from remote peer 256f9de1d4de4561b4f186af956880ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:07.283396 1815 raft_consensus.cc:1261] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 271d193fc4314739b335db51778d5ffc [term 1 FOLLOWER]: Refusing update from remote peer 256f9de1d4de4561b4f186af956880ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:07.284132 2050 consensus_queue.cc:1035] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 256f9de1d4de4561b4f186af956880ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "173ed386c375473890ba5f27ebe29d22" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260620 02:52:07.285210 2050 consensus_queue.cc:1035] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 256f9de1d4de4561b4f186af956880ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "271d193fc4314739b335db51778d5ffc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:52:07.290908 2066 mvcc.cc:205] Tried to move back new op lower bound from 7298760406144368640 to 7298760404203601920. Current Snapshot: MvccSnapshot[applied={T|T < 7298760406144368640}]
I20260620 02:52:07.310154 2067 mvcc.cc:205] Tried to move back new op lower bound from 7298760406144368640 to 7298760404203601920. Current Snapshot: MvccSnapshot[applied={T|T < 7298760406144368640}]
I20260620 02:52:07.334299 2086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:07.325859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40121 (local address 127.0.97.2:44031)
0620 02:52:07.326899 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:07.326923 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:07.327090 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:07.327479 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:07.327485 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:07.327500 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:52:07.327993 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:07.328007 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:07.329398 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:07.329405 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:07.331540 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:07.331557 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:07.332687 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:07.332695 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:07.333082 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:07.333175 (+ 93us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:52:07.333718 (+ 543us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:52:07.333731 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 02:52:07.333830 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 02:52:07.333968 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"thread_start_us":876,"threads_started":1}
I20260620 02:52:07.334285 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:07.326322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44031 (local address 127.0.97.3:40121)
0620 02:52:07.326519 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:07.326527 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:07.327009 (+ 482us) server_negotiation.cc:407] Connection header received
0620 02:52:07.327222 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:07.327227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:07.327335 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:07.327429 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:52:07.328202 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:07.329146 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:07.331767 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:07.332446 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:07.332611 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:07.333387 (+ 776us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:52:07.333517 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:52:07.333721 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:07.333882 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 02:52:07.333976 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 02:52:07.392522 1513 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:52:07.392796 1513 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:52:07.396366 1513 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32de122f2423469682096ed5bb608fa8: Sending DeleteTablet for 3 replicas of tablet a90d08e6f2ff4ba09e9a7ddc454585f0
I20260620 02:52:07.397215 1651 tablet_service.cc:1514] Processing DeleteTablet for tablet a90d08e6f2ff4ba09e9a7ddc454585f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:07 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48110
I20260620 02:52:07.397406 1789 tablet_service.cc:1514] Processing DeleteTablet for tablet a90d08e6f2ff4ba09e9a7ddc454585f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:07 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47426
I20260620 02:52:07.398175 1929 tablet_service.cc:1514] Processing DeleteTablet for tablet a90d08e6f2ff4ba09e9a7ddc454585f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:07 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52906
I20260620 02:52:07.409988 2089 tablet_replica.cc:330] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 173ed386c375473890ba5f27ebe29d22: stopping tablet replica
I20260620 02:52:07.410791 2089 raft_consensus.cc:2229] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 173ed386c375473890ba5f27ebe29d22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:52:07.411121 2089 raft_consensus.cc:2258] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 173ed386c375473890ba5f27ebe29d22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:07.412407 2089 ts_tablet_manager.cc:1905] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 173ed386c375473890ba5f27ebe29d22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:07.424394 2091 tablet_replica.cc:330] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 256f9de1d4de4561b4f186af956880ea: stopping tablet replica
I20260620 02:52:07.424800 2091 raft_consensus.cc:2229] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 256f9de1d4de4561b4f186af956880ea [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:52:07.425534 2091 raft_consensus.cc:2258] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 256f9de1d4de4561b4f186af956880ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:07.426674 2090 tablet_replica.cc:330] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 271d193fc4314739b335db51778d5ffc: stopping tablet replica
I20260620 02:52:07.426939 2090 raft_consensus.cc:2229] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 271d193fc4314739b335db51778d5ffc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:52:07.427290 2090 raft_consensus.cc:2258] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 271d193fc4314739b335db51778d5ffc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:07.428525 2090 ts_tablet_manager.cc:1905] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 271d193fc4314739b335db51778d5ffc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:07.431962 2091 ts_tablet_manager.cc:1905] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 256f9de1d4de4561b4f186af956880ea: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 02:52:07 dist-test-slave-8p8c krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923926, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:52:07.452171 2090 ts_tablet_manager.cc:1918] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 271d193fc4314739b335db51778d5ffc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:52:07.453104 2090 log.cc:1196] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 271d193fc4314739b335db51778d5ffc: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-1/wal/wals/a90d08e6f2ff4ba09e9a7ddc454585f0
I20260620 02:52:07.453933 2090 ts_tablet_manager.cc:1939] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 271d193fc4314739b335db51778d5ffc: Deleting consensus metadata
I20260620 02:52:07.462127 2093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:07.417232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48120 (local address 127.0.97.1:40297)
0620 02:52:07.426096 (+ 8864us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:07.426103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:07.431071 (+ 4968us) server_negotiation.cc:407] Connection header received
0620 02:52:07.431319 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:07.431325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:07.431413 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:07.431500 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:07.433539 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:07.434772 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:07.436806 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:07.437483 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:07.437629 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:07.443458 (+ 5829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:07.443479 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:07.443501 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:07.443551 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:07.456468 (+ 12917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:07.457710 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:07.457723 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:07.457727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:07.457933 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:07.458458 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:07.458466 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:07.458483 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:07.458963 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:07.459298 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:07.460546 (+ 1248us) server_negotiation.cc:299] Negotiation successful
0620 02:52:07.460806 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8716,"thread_start_us":8612,"threads_started":1}
W20260620 02:52:07.463357 1716 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48120
I20260620 02:52:07.471830 1499 catalog_manager.cc:4841] TS 271d193fc4314739b335db51778d5ffc (127.0.97.2:33681): tablet a90d08e6f2ff4ba09e9a7ddc454585f0 (table test-table [id=904ddbe955764c37a1436cb9df5b5fbc]) successfully deleted
I20260620 02:52:07.501847 2096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:07.468744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46242 (local address 127.0.97.62:38391)
0620 02:52:07.482872 (+ 14128us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:07.482880 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:07.482909 (+ 29us) server_negotiation.cc:407] Connection header received
0620 02:52:07.482987 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:07.482994 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:07.483108 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:07.483260 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:07.485673 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:07.486999 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:07.489282 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:07.489998 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:07.490157 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:07.493507 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:07.493514 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:07.493520 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:07.493591 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:07.498343 (+ 4752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:07.499330 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:07.499336 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:07.499340 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:07.499494 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:07.499965 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:07.499971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:07.499974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:07.500332 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:07.500584 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:07.501176 (+ 592us) server_negotiation.cc:299] Negotiation successful
0620 02:52:07.501440 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13966,"thread_start_us":13764,"threads_started":1}
W20260620 02:52:07.502883 1513 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46242
I20260620 02:52:07.506639 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 1585
I20260620 02:52:07.515096 2089 ts_tablet_manager.cc:1918] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 173ed386c375473890ba5f27ebe29d22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:52:07.515291 2089 log.cc:1196] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 173ed386c375473890ba5f27ebe29d22: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-0/wal/wals/a90d08e6f2ff4ba09e9a7ddc454585f0
I20260620 02:52:07.515832 2089 ts_tablet_manager.cc:1939] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 173ed386c375473890ba5f27ebe29d22: Deleting consensus metadata
I20260620 02:52:07.517242 2091 ts_tablet_manager.cc:1918] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 256f9de1d4de4561b4f186af956880ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:52:07.517426 2091 log.cc:1196] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 256f9de1d4de4561b4f186af956880ea: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781923914418765-388-0/minicluster-data/ts-2/wal/wals/a90d08e6f2ff4ba09e9a7ddc454585f0
I20260620 02:52:07.517941 2091 ts_tablet_manager.cc:1939] T a90d08e6f2ff4ba09e9a7ddc454585f0 P 256f9de1d4de4561b4f186af956880ea: Deleting consensus metadata
I20260620 02:52:07.519886 1500 catalog_manager.cc:4841] TS 173ed386c375473890ba5f27ebe29d22 (127.0.97.1:40297): tablet a90d08e6f2ff4ba09e9a7ddc454585f0 (table test-table [id=904ddbe955764c37a1436cb9df5b5fbc]) successfully deleted
I20260620 02:52:07.520265 1499 catalog_manager.cc:4841] TS 256f9de1d4de4561b4f186af956880ea (127.0.97.3:40121): tablet a90d08e6f2ff4ba09e9a7ddc454585f0 (table test-table [id=904ddbe955764c37a1436cb9df5b5fbc]) successfully deleted
I20260620 02:52:07.531153 2093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:07.508705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48122 (local address 127.0.97.1:40297)
0620 02:52:07.508919 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:07.508925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:07.513487 (+ 4562us) server_negotiation.cc:407] Connection header received
0620 02:52:07.513674 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:07.513681 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:07.513813 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:07.513937 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:07.516018 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:07.517161 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:07.519256 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:07.519963 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:07.520074 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:07.523569 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:07.523577 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:07.523583 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:07.523628 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:07.527876 (+ 4248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:07.528796 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:07.528802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:07.528807 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:07.528949 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:07.529391 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:07.529397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:07.529404 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:07.529751 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:07.529980 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:07.530564 (+ 584us) server_negotiation.cc:299] Negotiation successful
0620 02:52:07.530802 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 02:52:07.532543 1716 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48122
W20260620 02:52:07.533138 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 1585: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:52:07.533344 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 1585
I20260620 02:52:07.560097 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 1725
Jun 20 02:52:07 dist-test-slave-8p8c krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923926, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:52:07.580577 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:07.561554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47448 (local address 127.0.97.2:33681)
0620 02:52:07.561758 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:07.561764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:07.561785 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:52:07.561847 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:07.561853 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:07.561932 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:07.562042 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:07.564012 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:07.565325 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:07.567554 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:07.568338 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:07.568497 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:07.573943 (+ 5446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:07.573957 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:07.573975 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:07.574014 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:07.577330 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:07.578114 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:07.578122 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:07.578125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:07.578279 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:07.578766 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:07.578779 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:07.578788 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:07.579181 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:07.579521 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:07.579826 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 02:52:07.580087 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 02:52:07.581282 1855 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47448
W20260620 02:52:07.582018 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 1725: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:52:07.582222 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 1725
I20260620 02:52:07.606244 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 1864
Jun 20 02:52:07 dist-test-slave-8p8c krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923926, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:52:07.624325 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:07.607100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52924 (local address 127.0.97.3:40121)
0620 02:52:07.607306 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:07.607312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:07.607334 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:52:07.607496 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:07.607502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:07.607613 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:07.607734 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:07.609095 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:07.610289 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:07.611629 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:07.612338 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:07.612597 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:07.617282 (+ 4685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:07.617304 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:07.617334 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:07.617391 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:07.621472 (+ 4081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:07.622200 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:07.622207 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:07.622209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:07.622355 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:07.622774 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:07.622780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:07.622783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:07.623147 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:07.623467 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:07.623743 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 02:52:07.623967 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 02:52:07.625156 1995 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52924
W20260620 02:52:07.625887 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 1864: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:52:07.626055 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 1864
I20260620 02:52:07.645717 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 1482
I20260620 02:52:07.646922 1554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:07.760751 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 1482
2026-06-20T02:52:07Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4975 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:52:07 dist-test-slave-8p8c krb5kdc[2102](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:52:07 dist-test-slave-8p8c krb5kdc[2102](info): set up 2 sockets
Jun 20 02:52:07 dist-test-slave-8p8c krb5kdc[2102](info): commencing operation
krb5kdc: starting...
W20260620 02:52:08.886574 388 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:52:09 dist-test-slave-8p8c krb5kdc[2102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:52:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:52:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:52:09.095623 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45275
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:09.421725 2115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:09.422037 2115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:09.422192 2115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:09.429572 2115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:52:09.429675 2115 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 02:52:09.429754 2115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:09.429867 2115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:09.429967 2115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:52:09.430070 2115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:52:09.440016 2115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45275
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:09.441344 2115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:09.442970 2115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:09.455619 2121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:09.455590 2123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:09.455858 2120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:09.456365 2115 server_base.cc:1034] running on GCE node
I20260620 02:52:09.457175 2115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:09.458659 2115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:09.459961 2115 hybrid_clock.cc:648] HybridClock initialized: now 1781923929459873 us; error 68 us; skew 500 ppm
I20260620 02:52:09.460350 2115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:09 dist-test-slave-8p8c krb5kdc[2102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:09.463773 2115 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:52:09.465613 2115 webserver.cc:460] Webserver started at http://127.0.97.62:37465/ using document root <none> and password file <none>
I20260620 02:52:09.466413 2115 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:09.466555 2115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:09.466893 2115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:09.469864 2115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "5e40c9788c5d45278f7b8b3f8a7f7125"
format_stamp: "Formatted at 2026-06-20 02:52:09 on dist-test-slave-8p8c"
server_key: "8f48af02232e263c047be7c300cfeff1"
server_key_iv: "1fbd0d4bf0335d136d1a5fb2579090ba"
server_key_version: "encryptionkey@0"
I20260620 02:52:09.470715 2115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "5e40c9788c5d45278f7b8b3f8a7f7125"
format_stamp: "Formatted at 2026-06-20 02:52:09 on dist-test-slave-8p8c"
server_key: "8f48af02232e263c047be7c300cfeff1"
server_key_iv: "1fbd0d4bf0335d136d1a5fb2579090ba"
server_key_version: "encryptionkey@0"
I20260620 02:52:09.475850 2115 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 02:52:09.479423 2130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:09.480862 2115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 02:52:09.481061 2115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "5e40c9788c5d45278f7b8b3f8a7f7125"
format_stamp: "Formatted at 2026-06-20 02:52:09 on dist-test-slave-8p8c"
server_key: "8f48af02232e263c047be7c300cfeff1"
server_key_iv: "1fbd0d4bf0335d136d1a5fb2579090ba"
server_key_version: "encryptionkey@0"
I20260620 02:52:09.481284 2115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:09.500984 2115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:09.505455 2115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:09.505723 2115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:09.528617 2115 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45275
I20260620 02:52:09.528635 2191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45275 every 8 connection(s)
I20260620 02:52:09.529965 2115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:52:09.534134 2192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:09.538576 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 2115
I20260620 02:52:09.538810 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:52:09.539505 388 external_mini_cluster.cc:1428] Setting key a562852809040c162e51cde92ae5c5db
I20260620 02:52:09.545050 2192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e40c9788c5d45278f7b8b3f8a7f7125: Bootstrap starting.
I20260620 02:52:09.549010 2192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e40c9788c5d45278f7b8b3f8a7f7125: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:09.550293 2192 log.cc:824] T 00000000000000000000000000000000 P 5e40c9788c5d45278f7b8b3f8a7f7125: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:09.553570 2192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e40c9788c5d45278f7b8b3f8a7f7125: No bootstrap required, opened a new log
Jun 20 02:52:09 dist-test-slave-8p8c krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:09.564357 2192 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e40c9788c5d45278f7b8b3f8a7f7125 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e40c9788c5d45278f7b8b3f8a7f7125" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45275 } }
I20260620 02:52:09.564816 2195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:09.541636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57066 (local address 127.0.97.62:45275)
0620 02:52:09.542522 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:09.542534 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:09.542577 (+ 43us) server_negotiation.cc:407] Connection header received
0620 02:52:09.544293 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:09.544317 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:09.544721 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:09.545230 (+ 509us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:09.546973 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:09.548374 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:09.550214 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:09.551263 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:09.551408 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:09.555667 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:09.555679 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:09.555698 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:09.555752 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:09.560097 (+ 4345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:09.560804 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:09.560818 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:09.560829 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:09.561045 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:09.561412 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:09.561418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:09.561422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:09.562157 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:09.562481 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:09.562967 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 02:52:09.563444 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":379,"threads_started":1}
I20260620 02:52:09.564842 2192 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e40c9788c5d45278f7b8b3f8a7f7125 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:09.564958 2192 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e40c9788c5d45278f7b8b3f8a7f7125 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e40c9788c5d45278f7b8b3f8a7f7125, State: Initialized, Role: FOLLOWER
I20260620 02:52:09.565593 2192 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e40c9788c5d45278f7b8b3f8a7f7125 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e40c9788c5d45278f7b8b3f8a7f7125" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45275 } }
I20260620 02:52:09.565850 2192 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e40c9788c5d45278f7b8b3f8a7f7125 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:52:09.565992 2192 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e40c9788c5d45278f7b8b3f8a7f7125 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:52:09.566180 2192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e40c9788c5d45278f7b8b3f8a7f7125 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:09.569034 2192 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e40c9788c5d45278f7b8b3f8a7f7125 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e40c9788c5d45278f7b8b3f8a7f7125" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45275 } }
I20260620 02:52:09.569479 2192 leader_election.cc:302] T 00000000000000000000000000000000 P 5e40c9788c5d45278f7b8b3f8a7f7125 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e40c9788c5d45278f7b8b3f8a7f7125; no voters:
I20260620 02:52:09.570086 2192 leader_election.cc:288] T 00000000000000000000000000000000 P 5e40c9788c5d45278f7b8b3f8a7f7125 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:52:09.570293 2197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e40c9788c5d45278f7b8b3f8a7f7125 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:09.571156 2197 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e40c9788c5d45278f7b8b3f8a7f7125 [term 1 LEADER]: Becoming Leader. State: Replica: 5e40c9788c5d45278f7b8b3f8a7f7125, State: Running, Role: LEADER
I20260620 02:52:09.571740 2197 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e40c9788c5d45278f7b8b3f8a7f7125 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e40c9788c5d45278f7b8b3f8a7f7125" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45275 } }
I20260620 02:52:09.572489 2192 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e40c9788c5d45278f7b8b3f8a7f7125 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:52:09.574409 2198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e40c9788c5d45278f7b8b3f8a7f7125 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e40c9788c5d45278f7b8b3f8a7f7125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e40c9788c5d45278f7b8b3f8a7f7125" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45275 } } }
I20260620 02:52:09.574594 2199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e40c9788c5d45278f7b8b3f8a7f7125 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e40c9788c5d45278f7b8b3f8a7f7125. Latest consensus state: current_term: 1 leader_uuid: "5e40c9788c5d45278f7b8b3f8a7f7125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e40c9788c5d45278f7b8b3f8a7f7125" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45275 } } }
I20260620 02:52:09.574873 2199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e40c9788c5d45278f7b8b3f8a7f7125 [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:09.574873 2198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e40c9788c5d45278f7b8b3f8a7f7125 [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:09.575989 2204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:52:09.586843 2204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:52:09.597666 2204 catalog_manager.cc:1347] Generated new cluster ID: b0cd642251fd47bb946bb33fed1ad612
I20260620 02:52:09.597818 2204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:52:09.606740 2204 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:52:09.608242 2204 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:52:09.619731 2204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e40c9788c5d45278f7b8b3f8a7f7125: Generated new TSK 0
I20260620 02:52:09.620960 2204 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:52:09.676162 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45275
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:10.007278 2218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:10.007586 2218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:10.007687 2218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:10.015054 2218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:10.015163 2218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:10.015247 2218 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 02:52:10.015298 2218 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:52:10.015359 2218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:52:10.025112 2218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:45275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:10.026347 2218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:10.027685 2218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:10.040871 2223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:10.040956 2224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:10.040980 2226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:10.042122 2218 server_base.cc:1034] running on GCE node
I20260620 02:52:10.042857 2218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:10.043771 2218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:10.045034 2218 hybrid_clock.cc:648] HybridClock initialized: now 1781923930044938 us; error 86 us; skew 500 ppm
I20260620 02:52:10.045377 2218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:10 dist-test-slave-8p8c krb5kdc[2102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:10.048590 2218 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:52:10.050071 2218 webserver.cc:460] Webserver started at http://127.0.97.1:33871/ using document root <none> and password file <none>
I20260620 02:52:10.050815 2218 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:10.050915 2218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:10.051167 2218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:10.054147 2218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-0/data/instance:
uuid: "9f368a6470774802a059bea680da7b20"
format_stamp: "Formatted at 2026-06-20 02:52:10 on dist-test-slave-8p8c"
server_key: "a27ab506d0e46db57077b43b170a70d2"
server_key_iv: "7bb0b55cb2a496e10a6352c446d3b704"
server_key_version: "encryptionkey@0"
I20260620 02:52:10.054925 2218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-0/wal/instance:
uuid: "9f368a6470774802a059bea680da7b20"
format_stamp: "Formatted at 2026-06-20 02:52:10 on dist-test-slave-8p8c"
server_key: "a27ab506d0e46db57077b43b170a70d2"
server_key_iv: "7bb0b55cb2a496e10a6352c446d3b704"
server_key_version: "encryptionkey@0"
I20260620 02:52:10.060297 2218 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:52:10.063931 2233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:10.065377 2218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:52:10.065564 2218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-0/wal
uuid: "9f368a6470774802a059bea680da7b20"
format_stamp: "Formatted at 2026-06-20 02:52:10 on dist-test-slave-8p8c"
server_key: "a27ab506d0e46db57077b43b170a70d2"
server_key_iv: "7bb0b55cb2a496e10a6352c446d3b704"
server_key_version: "encryptionkey@0"
I20260620 02:52:10.065764 2218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:10.080636 2218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:10.084831 2218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:10.085127 2218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:10.087813 2218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:10.087942 2218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:10.088053 2218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:10.088145 2218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:52:10 dist-test-slave-8p8c krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:10.112059 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:10.088516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59673 (local address 127.0.97.62:45275)
0620 02:52:10.089075 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:10.089080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:10.090294 (+ 1214us) server_negotiation.cc:407] Connection header received
0620 02:52:10.092569 (+ 2275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:10.092576 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:10.092675 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:10.092819 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:10.095157 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:10.096189 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:10.098708 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:10.099310 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:10.099451 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:10.104819 (+ 5368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:10.104827 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:10.104834 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:10.104872 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:10.108136 (+ 3264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:10.108911 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:10.108918 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:10.108923 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:10.109086 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:10.109641 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:10.109652 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:10.109655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:10.110014 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:10.110216 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:10.111216 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0620 02:52:10.111500 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":360,"threads_started":1}
I20260620 02:52:10.112794 2243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:10.089182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45275 (local address 127.0.97.1:59673)
0620 02:52:10.090019 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:10.090071 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:10.092235 (+ 2164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:10.093168 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:10.093175 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:10.093942 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:10.094965 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:10.094995 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:10.096355 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:10.096363 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:10.098569 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:10.098582 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:10.099565 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:10.099574 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:10.099793 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:10.100702 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:10.100724 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:10.104529 (+ 3805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:10.108269 (+ 3740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:10.108278 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:10.108294 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:10.108791 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:10.109232 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:10.109236 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:10.109240 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:10.109518 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:10.110198 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:10.110209 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:10.110748 (+ 539us) client_negotiation.cc:769] Sending connection context
0620 02:52:10.111094 (+ 346us) client_negotiation.cc:240] Negotiation successful
0620 02:52:10.111590 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":383,"threads_started":1}
I20260620 02:52:10.134433 2218 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44893
I20260620 02:52:10.134482 2353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44893 every 8 connection(s)
I20260620 02:52:10.135720 2218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:52:10.136022 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:10.119677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59229 (local address 127.0.97.62:45275)
0620 02:52:10.119928 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:10.119936 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:10.121167 (+ 1231us) server_negotiation.cc:407] Connection header received
0620 02:52:10.121352 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:10.121356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:10.121414 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:10.121554 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:10.123045 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:10.124062 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:10.126513 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:10.127054 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:10.127188 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:10.130194 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:10.130200 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:10.130207 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:10.130248 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:10.133068 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:10.133963 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:10.133970 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:10.133973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:10.134146 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:10.134667 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:10.134672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:10.134675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:10.134925 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:10.135087 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:10.135457 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 02:52:10.135677 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 02:52:10.136163 2322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:10.119779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45275 (local address 127.0.97.1:59229)
0620 02:52:10.121036 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:10.121061 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:10.121232 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:10.121557 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:10.121565 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:10.122305 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:10.122820 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:10.122830 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:10.124199 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:10.124209 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:10.126320 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:10.126337 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:10.127308 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:10.127317 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:10.127555 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:10.128248 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:10.128278 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:10.129989 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:10.133192 (+ 3203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:10.133202 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:10.133206 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:10.133847 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:10.134293 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:10.134299 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:10.134306 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:10.134515 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:10.135051 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:10.135056 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:10.135225 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 02:52:10.135357 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 02:52:10.135674 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1061,"thread_start_us":475,"threads_started":1}
I20260620 02:52:10.138100 2354 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45275
I20260620 02:52:10.138356 2354 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:10.139124 2354 heartbeater.cc:511] Master 127.0.97.62:45275 requested a full tablet report, sending...
I20260620 02:52:10.141136 2147 ts_manager.cc:194] Registered new tserver with Master: 9f368a6470774802a059bea680da7b20 (127.0.97.1:44893)
I20260620 02:52:10.142652 2147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59673
I20260620 02:52:10.144550 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 2218
I20260620 02:52:10.144742 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:52:10.145545 388 external_mini_cluster.cc:1428] Setting key 88509f2cface479f5a5d9e113d205af8
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:52:10.186254 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45275
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:10.507712 2358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:10.508009 2358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:10.508176 2358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:10.516402 2358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:10.516516 2358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:10.516682 2358 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 02:52:10.516753 2358 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:52:10.516861 2358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:52:10.527776 2358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:45275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:10.529220 2358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:10.530627 2358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:10.546231 2363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:10.546260 2366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:10.546240 2364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:10.547145 2358 server_base.cc:1034] running on GCE node
I20260620 02:52:10.547786 2358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:10.548965 2358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:10.550221 2358 hybrid_clock.cc:648] HybridClock initialized: now 1781923930550177 us; error 36 us; skew 500 ppm
I20260620 02:52:10.550618 2358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:10 dist-test-slave-8p8c krb5kdc[2102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:10.554145 2358 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:52:10.555751 2358 webserver.cc:460] Webserver started at http://127.0.97.2:35153/ using document root <none> and password file <none>
I20260620 02:52:10.556411 2358 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:10.556521 2358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:10.556772 2358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:10.559679 2358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-1/data/instance:
uuid: "8c0a4114184647fb9fa556aec7f269dd"
format_stamp: "Formatted at 2026-06-20 02:52:10 on dist-test-slave-8p8c"
server_key: "8f819ae7183a09f54f9388bc32a9c7f9"
server_key_iv: "e7cf880c6c0b74156740f9a31132bca5"
server_key_version: "encryptionkey@0"
I20260620 02:52:10.560382 2358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8c0a4114184647fb9fa556aec7f269dd"
format_stamp: "Formatted at 2026-06-20 02:52:10 on dist-test-slave-8p8c"
server_key: "8f819ae7183a09f54f9388bc32a9c7f9"
server_key_iv: "e7cf880c6c0b74156740f9a31132bca5"
server_key_version: "encryptionkey@0"
I20260620 02:52:10.565444 2358 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:52:10.568847 2373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:10.570482 2358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:52:10.570660 2358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-1/wal
uuid: "8c0a4114184647fb9fa556aec7f269dd"
format_stamp: "Formatted at 2026-06-20 02:52:10 on dist-test-slave-8p8c"
server_key: "8f819ae7183a09f54f9388bc32a9c7f9"
server_key_iv: "e7cf880c6c0b74156740f9a31132bca5"
server_key_version: "encryptionkey@0"
I20260620 02:52:10.570865 2358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:10.593372 2358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:10.597523 2358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:10.597821 2358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:10.600512 2358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:10.600606 2358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:10.600790 2358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:10.600903 2358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:52:10 dist-test-slave-8p8c krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:10.627442 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:10.601552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60895 (local address 127.0.97.62:45275)
0620 02:52:10.601770 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:10.601778 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:10.603498 (+ 1720us) server_negotiation.cc:407] Connection header received
0620 02:52:10.606911 (+ 3413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:10.606918 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:10.607015 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:10.607148 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:10.609438 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:10.610668 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:10.613577 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:10.614299 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:10.614511 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:10.619255 (+ 4744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:10.619265 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:10.619274 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:10.619315 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:10.623517 (+ 4202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:10.624581 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:10.624589 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:10.624596 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:10.624771 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:10.625229 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:10.625236 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:10.625242 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:10.625706 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:10.625905 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:10.626854 (+ 949us) server_negotiation.cc:299] Negotiation successful
0620 02:52:10.627095 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:52:10.629242 2382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:10.602265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45275 (local address 127.0.97.2:60895)
0620 02:52:10.603308 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:10.603366 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:10.606518 (+ 3152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:10.607417 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:10.607426 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:10.608279 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:10.609262 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:10.609284 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:10.610905 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:10.610914 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:10.613348 (+ 2434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:10.613370 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:10.614617 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:10.614629 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:10.615014 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:10.615888 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:10.615934 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:10.619015 (+ 3081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:10.623727 (+ 4712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:10.623743 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:10.623765 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:10.624437 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:10.624923 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:10.624927 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:10.624930 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:10.625115 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:10.625904 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:10.625910 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:10.626291 (+ 381us) client_negotiation.cc:769] Sending connection context
0620 02:52:10.626770 (+ 479us) client_negotiation.cc:240] Negotiation successful
0620 02:52:10.627343 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":390,"threads_started":1}
I20260620 02:52:10.657796 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:10.636981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58325 (local address 127.0.97.62:45275)
0620 02:52:10.637197 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:10.637204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:10.638810 (+ 1606us) server_negotiation.cc:407] Connection header received
0620 02:52:10.639036 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:10.639051 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:10.639147 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:10.639256 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:10.640784 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:10.641510 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:10.644126 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:10.644798 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:10.644962 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:10.650821 (+ 5859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:10.650829 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:10.650835 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:10.650883 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:10.654034 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:10.655147 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:10.655155 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:10.655161 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:10.655325 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:10.656003 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:10.656011 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:10.656017 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:10.656566 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:10.656742 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:10.657214 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 02:52:10.657440 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 02:52:10.658082 2445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:10.637143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45275 (local address 127.0.97.2:58325)
0620 02:52:10.638701 (+ 1558us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:10.638718 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:10.638888 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:10.639339 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:10.639346 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:10.640147 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:10.640605 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:10.640622 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:10.641652 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:10.641660 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:10.643868 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:10.643888 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:10.645133 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:10.645148 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:10.645486 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:10.646449 (+ 963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:10.646478 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:10.650530 (+ 4052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:10.654205 (+ 3675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:10.654212 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:10.654217 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:10.654983 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:10.655452 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:10.655458 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:10.655461 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:10.655836 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:10.656710 (+ 874us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:10.656717 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:10.656909 (+ 192us) client_negotiation.cc:769] Sending connection context
0620 02:52:10.657085 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 02:52:10.657469 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1434,"thread_start_us":530,"threads_started":1}
I20260620 02:52:10.662994 2358 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36181
I20260620 02:52:10.663308 2492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36181 every 8 connection(s)
I20260620 02:52:10.664482 2358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:52:10.664793 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 2358
I20260620 02:52:10.664961 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:52:10.665797 388 external_mini_cluster.cc:1428] Setting key a5abb0cd321023df65b9a2961883edd3
I20260620 02:52:10.666818 2493 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45275
I20260620 02:52:10.667125 2493 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:10.667910 2493 heartbeater.cc:511] Master 127.0.97.62:45275 requested a full tablet report, sending...
I20260620 02:52:10.669386 2147 ts_manager.cc:194] Registered new tserver with Master: 8c0a4114184647fb9fa556aec7f269dd (127.0.97.2:36181)
I20260620 02:52:10.670276 2147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60895
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:52:10.714710 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45275
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:11.045956 2497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:11.046298 2497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:11.046489 2497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:11.054194 2497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:11.054359 2497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:11.054567 2497 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 02:52:11.054656 2497 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:52:11.054775 2497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:52:11.064653 2497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:45275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:11.066004 2497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:11.067814 2497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:11.081388 2503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:11.081573 2502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:11.081535 2505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:11.082271 2497 server_base.cc:1034] running on GCE node
I20260620 02:52:11.082885 2497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:11.083776 2497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:11.085011 2497 hybrid_clock.cc:648] HybridClock initialized: now 1781923931084942 us; error 56 us; skew 500 ppm
I20260620 02:52:11.085384 2497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:11 dist-test-slave-8p8c krb5kdc[2102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:11.088601 2497 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:52:11.090040 2497 webserver.cc:460] Webserver started at http://127.0.97.3:38715/ using document root <none> and password file <none>
I20260620 02:52:11.090757 2497 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:11.090862 2497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:11.091101 2497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:11.093948 2497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-2/data/instance:
uuid: "45e33b6b3f894bc884f6c093deed65ea"
format_stamp: "Formatted at 2026-06-20 02:52:11 on dist-test-slave-8p8c"
server_key: "bef28f8df95a4a59ca9c6338127d3056"
server_key_iv: "28f11ea46fcac4fa5627d6d1540756aa"
server_key_version: "encryptionkey@0"
I20260620 02:52:11.094726 2497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-2/wal/instance:
uuid: "45e33b6b3f894bc884f6c093deed65ea"
format_stamp: "Formatted at 2026-06-20 02:52:11 on dist-test-slave-8p8c"
server_key: "bef28f8df95a4a59ca9c6338127d3056"
server_key_iv: "28f11ea46fcac4fa5627d6d1540756aa"
server_key_version: "encryptionkey@0"
I20260620 02:52:11.099705 2497 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 02:52:11.103111 2512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:11.104532 2497 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:52:11.104715 2497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-2/wal
uuid: "45e33b6b3f894bc884f6c093deed65ea"
format_stamp: "Formatted at 2026-06-20 02:52:11 on dist-test-slave-8p8c"
server_key: "bef28f8df95a4a59ca9c6338127d3056"
server_key_iv: "28f11ea46fcac4fa5627d6d1540756aa"
server_key_version: "encryptionkey@0"
I20260620 02:52:11.104918 2497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:11.124239 2497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:11.128630 2497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:11.128932 2497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:11.131961 2497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:11.132081 2497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:11.132194 2497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:11.132292 2497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:11.145830 2354 heartbeater.cc:503] Master 127.0.97.62:45275 was elected leader, sending a full tablet report...
Jun 20 02:52:11 dist-test-slave-8p8c krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:11.162524 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:11.132635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56083 (local address 127.0.97.62:45275)
0620 02:52:11.132915 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:11.132923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:11.134643 (+ 1720us) server_negotiation.cc:407] Connection header received
0620 02:52:11.137533 (+ 2890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:11.137539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:11.137626 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:11.137697 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:11.141028 (+ 3331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.142408 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.145751 (+ 3343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.146687 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.146820 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:11.153059 (+ 6239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:11.153068 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:11.153075 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:11.153130 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:11.157371 (+ 4241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:11.158956 (+ 1585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:11.158962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:11.158965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:11.159116 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:11.159761 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:11.159769 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:11.159774 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:11.160235 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:11.160377 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:11.161770 (+ 1393us) server_negotiation.cc:299] Negotiation successful
0620 02:52:11.162052 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 02:52:11.164223 2521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:11.133230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45275 (local address 127.0.97.3:56083)
0620 02:52:11.134433 (+ 1203us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:11.134484 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:11.137169 (+ 2685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:11.138322 (+ 1153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:11.138334 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:11.139372 (+ 1038us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:11.140713 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:11.140750 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.142590 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.142594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:11.145534 (+ 2940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:11.145559 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.147266 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.147278 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:11.147676 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:11.148616 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:11.148658 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:11.152729 (+ 4071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:11.157557 (+ 4828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:11.157571 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:11.157603 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:11.158830 (+ 1227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:11.159241 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:11.159247 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:11.159253 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:11.159591 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:11.160495 (+ 904us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:11.160507 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:11.161208 (+ 701us) client_negotiation.cc:769] Sending connection context
0620 02:52:11.161630 (+ 422us) client_negotiation.cc:240] Negotiation successful
0620 02:52:11.162511 (+ 881us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":857,"thread_start_us":602,"threads_started":1}
I20260620 02:52:11.177695 2497 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36599
I20260620 02:52:11.177717 2631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36599 every 8 connection(s)
I20260620 02:52:11.179108 2497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:52:11.181557 2632 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45275
I20260620 02:52:11.181800 2632 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:11.182652 2632 heartbeater.cc:511] Master 127.0.97.62:45275 requested a full tablet report, sending...
I20260620 02:52:11.184046 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 2497
I20260620 02:52:11.184218 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:52:11.184684 388 external_mini_cluster.cc:1428] Setting key 94d8a5a7d3706073e0b6491238571a7c
I20260620 02:52:11.185621 2147 ts_manager.cc:194] Registered new tserver with Master: 45e33b6b3f894bc884f6c093deed65ea (127.0.97.3:36599)
I20260620 02:52:11.186800 2147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56083
I20260620 02:52:11.188081 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:52:11.189277 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:11.171770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37703 (local address 127.0.97.62:45275)
0620 02:52:11.171984 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:11.171993 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:11.175208 (+ 3215us) server_negotiation.cc:407] Connection header received
0620 02:52:11.175466 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:11.175470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:11.175552 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:11.175701 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:11.176893 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.177957 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.179715 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.180364 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.180494 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:11.182936 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:11.182942 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:11.182946 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:11.182998 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:11.186433 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:11.187241 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:11.187246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:11.187250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:11.187390 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:11.187756 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:11.187761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:11.187764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:11.188040 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:11.188190 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:11.188718 (+ 528us) server_negotiation.cc:299] Negotiation successful
0620 02:52:11.188920 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 02:52:11.189431 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:11.171771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45275 (local address 127.0.97.3:37703)
0620 02:52:11.175067 (+ 3296us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:11.175094 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:11.175288 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:11.175726 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:11.175731 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:11.176261 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:11.176694 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:11.176709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.178131 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.178135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:11.179523 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:11.179540 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.180733 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.180739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:11.180927 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:11.181612 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:11.181636 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:11.182726 (+ 1090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:11.186549 (+ 3823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:11.186556 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:11.186560 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:11.187110 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:11.187506 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:11.187511 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:11.187515 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:11.187656 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:11.188175 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:11.188180 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:11.188444 (+ 264us) client_negotiation.cc:769] Sending connection context
0620 02:52:11.188591 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 02:52:11.188915 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3080,"mutex_wait_us":80,"thread_start_us":2908,"threads_started":1}
Jun 20 02:52:11 dist-test-slave-8p8c krb5kdc[2102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:52:11 dist-test-slave-8p8c krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:11.257145 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:11.235262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57082 (local address 127.0.97.62:45275)
0620 02:52:11.235524 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:11.235531 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:11.235928 (+ 397us) server_negotiation.cc:407] Connection header received
0620 02:52:11.236160 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:11.236164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:11.236237 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:11.236314 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:11.238432 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.239171 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.241394 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.241794 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.241898 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:11.246852 (+ 4954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:11.246857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:11.246860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:11.246893 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:11.254540 (+ 7647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:11.255181 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:11.255186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:11.255197 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:11.255318 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:11.255696 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:11.255704 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:11.255707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:11.255978 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:11.256229 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:11.256556 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 02:52:11.256778 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 02:52:11.261210 2146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:52:11.271693 2146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:52:11.338965 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:11.288298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36181 (local address 127.0.0.1:49528)
0620 02:52:11.323122 (+ 34824us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:11.323151 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:11.323352 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:11.326096 (+ 2744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:11.326107 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:11.326157 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:11.326663 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:11.326680 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.329013 (+ 2333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.329021 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:11.331835 (+ 2814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:11.331852 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.337869 (+ 6017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.337877 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:11.338154 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:11.338235 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 02:52:11.338353 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 02:52:11.338574 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34612,"thread_start_us":43831,"threads_started":1}
I20260620 02:52:11.340155 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:11.289005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49528 (local address 127.0.97.2:36181)
0620 02:52:11.323932 (+ 34927us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:11.323945 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:11.323982 (+ 37us) server_negotiation.cc:407] Connection header received
0620 02:52:11.324111 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:11.324123 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:11.324454 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:11.324686 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:11.326916 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.328730 (+ 1814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.334626 (+ 5896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.336223 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.336347 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:11.336591 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:11.339356 (+ 2765us) server_negotiation.cc:299] Negotiation successful
0620 02:52:11.339544 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34695,"thread_start_us":45292,"threads_started":1}
I20260620 02:52:11.343954 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:11.289438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38826 (local address 127.0.97.3:36599)
0620 02:52:11.325451 (+ 36013us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:11.325463 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:11.332632 (+ 7169us) server_negotiation.cc:407] Connection header received
0620 02:52:11.332771 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:11.332779 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:11.333089 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:11.333313 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:11.334434 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.336454 (+ 2020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.339558 (+ 3104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.341504 (+ 1946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.341651 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:11.341907 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:11.343307 (+ 1400us) server_negotiation.cc:299] Negotiation successful
0620 02:52:11.343530 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35791,"thread_start_us":36294,"threads_started":1}
I20260620 02:52:11.344967 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:11.288415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36599 (local address 127.0.0.1:38826)
0620 02:52:11.332385 (+ 43970us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:11.332403 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:11.332514 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:11.333677 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:11.333685 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:11.333720 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:11.334143 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:11.334157 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.336662 (+ 2505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.336670 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:11.339351 (+ 2681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:11.339370 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.342066 (+ 2696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.342075 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:11.342296 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:11.342357 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 02:52:11.342474 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 02:52:11.344502 (+ 2028us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43855,"mutex_wait_us":1,"thread_start_us":37083,"threads_started":1}
I20260620 02:52:11.344496 2423 tablet_service.cc:1467] Processing CreateTablet for tablet dfcfe206467549feb7227bdb9101ab48 (DEFAULT_TABLE table=test-table [id=d299fc6e18f04127a0729f49915845cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:11.345947 2423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfcfe206467549feb7227bdb9101ab48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:11.347539 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:11.287761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44893 (local address 127.0.0.1:55274)
0620 02:52:11.294946 (+ 7185us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:11.294981 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:11.295163 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:11.337568 (+ 42405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:11.337573 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:11.337597 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:11.338090 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:11.338101 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.340998 (+ 2897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.341004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:11.343094 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:11.343108 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.346470 (+ 3362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.346479 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:11.346741 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:11.346792 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 02:52:11.346897 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 02:52:11.347069 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6930,"mutex_wait_us":23,"thread_start_us":6809,"threads_started":1}
I20260620 02:52:11.348716 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:11.287914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55274 (local address 127.0.97.1:44893)
0620 02:52:11.337033 (+ 49119us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:11.337040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:11.337063 (+ 23us) server_negotiation.cc:407] Connection header received
0620 02:52:11.337136 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:11.337141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:11.337377 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:11.337578 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:11.338303 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.340845 (+ 2542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.344010 (+ 3165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.345390 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.345533 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:11.345728 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:11.348137 (+ 2409us) server_negotiation.cc:299] Negotiation successful
0620 02:52:11.348364 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48959,"thread_start_us":48838,"threads_started":1}
I20260620 02:52:11.349056 2562 tablet_service.cc:1467] Processing CreateTablet for tablet dfcfe206467549feb7227bdb9101ab48 (DEFAULT_TABLE table=test-table [id=d299fc6e18f04127a0729f49915845cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:11.351253 2562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfcfe206467549feb7227bdb9101ab48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:11.352684 2284 tablet_service.cc:1467] Processing CreateTablet for tablet dfcfe206467549feb7227bdb9101ab48 (DEFAULT_TABLE table=test-table [id=d299fc6e18f04127a0729f49915845cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:11.354216 2284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfcfe206467549feb7227bdb9101ab48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:11.369972 2651 tablet_bootstrap.cc:492] T dfcfe206467549feb7227bdb9101ab48 P 8c0a4114184647fb9fa556aec7f269dd: Bootstrap starting.
I20260620 02:52:11.373564 2651 tablet_bootstrap.cc:654] T dfcfe206467549feb7227bdb9101ab48 P 8c0a4114184647fb9fa556aec7f269dd: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:11.374926 2651 log.cc:824] T dfcfe206467549feb7227bdb9101ab48 P 8c0a4114184647fb9fa556aec7f269dd: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:11.377400 2652 tablet_bootstrap.cc:492] T dfcfe206467549feb7227bdb9101ab48 P 9f368a6470774802a059bea680da7b20: Bootstrap starting.
I20260620 02:52:11.380045 2653 tablet_bootstrap.cc:492] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea: Bootstrap starting.
I20260620 02:52:11.382540 2652 tablet_bootstrap.cc:654] T dfcfe206467549feb7227bdb9101ab48 P 9f368a6470774802a059bea680da7b20: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:11.384035 2652 log.cc:824] T dfcfe206467549feb7227bdb9101ab48 P 9f368a6470774802a059bea680da7b20: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:11.385629 2653 tablet_bootstrap.cc:654] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:11.387295 2653 log.cc:824] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:11.387734 2651 tablet_bootstrap.cc:492] T dfcfe206467549feb7227bdb9101ab48 P 8c0a4114184647fb9fa556aec7f269dd: No bootstrap required, opened a new log
I20260620 02:52:11.388135 2651 ts_tablet_manager.cc:1397] T dfcfe206467549feb7227bdb9101ab48 P 8c0a4114184647fb9fa556aec7f269dd: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.003s
I20260620 02:52:11.393278 2652 tablet_bootstrap.cc:492] T dfcfe206467549feb7227bdb9101ab48 P 9f368a6470774802a059bea680da7b20: No bootstrap required, opened a new log
I20260620 02:52:11.393639 2652 ts_tablet_manager.cc:1397] T dfcfe206467549feb7227bdb9101ab48 P 9f368a6470774802a059bea680da7b20: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.011s
I20260620 02:52:11.400705 2651 raft_consensus.cc:348] T dfcfe206467549feb7227bdb9101ab48 P 8c0a4114184647fb9fa556aec7f269dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:11.401808 2651 raft_consensus.cc:374] T dfcfe206467549feb7227bdb9101ab48 P 8c0a4114184647fb9fa556aec7f269dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:11.401979 2651 raft_consensus.cc:729] T dfcfe206467549feb7227bdb9101ab48 P 8c0a4114184647fb9fa556aec7f269dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c0a4114184647fb9fa556aec7f269dd, State: Initialized, Role: FOLLOWER
I20260620 02:52:11.402906 2651 consensus_queue.cc:260] T dfcfe206467549feb7227bdb9101ab48 P 8c0a4114184647fb9fa556aec7f269dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:11.404260 2653 tablet_bootstrap.cc:492] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea: No bootstrap required, opened a new log
I20260620 02:52:11.404285 2652 raft_consensus.cc:348] T dfcfe206467549feb7227bdb9101ab48 P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:11.404621 2653 ts_tablet_manager.cc:1397] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.010s
I20260620 02:52:11.404775 2652 raft_consensus.cc:374] T dfcfe206467549feb7227bdb9101ab48 P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:11.404939 2652 raft_consensus.cc:729] T dfcfe206467549feb7227bdb9101ab48 P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f368a6470774802a059bea680da7b20, State: Initialized, Role: FOLLOWER
I20260620 02:52:11.405709 2652 consensus_queue.cc:260] T dfcfe206467549feb7227bdb9101ab48 P 9f368a6470774802a059bea680da7b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:11.410992 2652 ts_tablet_manager.cc:1428] T dfcfe206467549feb7227bdb9101ab48 P 9f368a6470774802a059bea680da7b20: Time spent starting tablet: real 0.017s user 0.013s sys 0.003s
I20260620 02:52:11.413424 2493 heartbeater.cc:503] Master 127.0.97.62:45275 was elected leader, sending a full tablet report...
I20260620 02:52:11.414116 2651 ts_tablet_manager.cc:1428] T dfcfe206467549feb7227bdb9101ab48 P 8c0a4114184647fb9fa556aec7f269dd: Time spent starting tablet: real 0.026s user 0.019s sys 0.004s
W20260620 02:52:11.417007 2494 tablet.cc:2369] T dfcfe206467549feb7227bdb9101ab48 P 8c0a4114184647fb9fa556aec7f269dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:52:11.420518 2653 raft_consensus.cc:348] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:11.420924 2653 raft_consensus.cc:374] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:11.421036 2653 raft_consensus.cc:729] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45e33b6b3f894bc884f6c093deed65ea, State: Initialized, Role: FOLLOWER
I20260620 02:52:11.421659 2653 consensus_queue.cc:260] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:11.422878 2632 heartbeater.cc:503] Master 127.0.97.62:45275 was elected leader, sending a full tablet report...
I20260620 02:52:11.423390 2653 ts_tablet_manager.cc:1428] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea: Time spent starting tablet: real 0.019s user 0.008s sys 0.005s
W20260620 02:52:11.431506 2633 tablet.cc:2369] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:52:11.561230 2659 raft_consensus.cc:482] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:52:11.561532 2659 raft_consensus.cc:504] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:11.563144 2659 leader_election.cc:288] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f368a6470774802a059bea680da7b20 (127.0.97.1:44893), 8c0a4114184647fb9fa556aec7f269dd (127.0.97.2:36181)
I20260620 02:52:11.570461 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:11.563678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55285 (local address 127.0.97.2:36181)
0620 02:52:11.564079 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:11.564089 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:11.564115 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:52:11.564209 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:11.564218 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:11.564297 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:11.564403 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:11.565007 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.566013 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.568333 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.569503 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.569649 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:11.569757 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:11.570030 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 02:52:11.570162 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218}
I20260620 02:52:11.570552 2521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:11.563667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36181 (local address 127.0.97.3:55285)
0620 02:52:11.563879 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:11.563899 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:11.564021 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:11.564451 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:11.564456 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:11.564502 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:11.564771 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:11.564779 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.566146 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.566150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:11.568162 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:11.568176 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.569707 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.569712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:11.569860 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:11.569887 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 02:52:11.569974 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 02:52:11.570071 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"spinlock_wait_cycles":23424}
I20260620 02:52:11.571162 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:11.563678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48891 (local address 127.0.97.1:44893)
0620 02:52:11.563868 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:11.563875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:11.564579 (+ 704us) server_negotiation.cc:407] Connection header received
0620 02:52:11.564709 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:11.564712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:11.564778 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:11.565025 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:11.565550 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.566508 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.568767 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.570094 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.570218 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:11.570344 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:11.570799 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 02:52:11.570927 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 02:52:11.571254 2660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:11.563680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44893 (local address 127.0.97.3:48891)
0620 02:52:11.564459 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:11.564475 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:11.564592 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:11.565072 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:11.565076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:11.565099 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:11.565368 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:11.565376 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.566650 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.566655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:11.568616 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:11.568625 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.570437 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.570456 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:11.570652 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:11.570682 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 02:52:11.570775 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 02:52:11.570902 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":67,"thread_start_us":425,"threads_started":1}
I20260620 02:52:11.571622 2448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfcfe206467549feb7227bdb9101ab48" candidate_uuid: "45e33b6b3f894bc884f6c093deed65ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c0a4114184647fb9fa556aec7f269dd" is_pre_election: true
I20260620 02:52:11.572104 2448 raft_consensus.cc:2454] T dfcfe206467549feb7227bdb9101ab48 P 8c0a4114184647fb9fa556aec7f269dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45e33b6b3f894bc884f6c093deed65ea in term 0.
I20260620 02:52:11.572245 2304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfcfe206467549feb7227bdb9101ab48" candidate_uuid: "45e33b6b3f894bc884f6c093deed65ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f368a6470774802a059bea680da7b20" is_pre_election: true
I20260620 02:52:11.572747 2304 raft_consensus.cc:2454] T dfcfe206467549feb7227bdb9101ab48 P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45e33b6b3f894bc884f6c093deed65ea in term 0.
I20260620 02:52:11.572840 2516 leader_election.cc:302] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45e33b6b3f894bc884f6c093deed65ea, 8c0a4114184647fb9fa556aec7f269dd; no voters:
I20260620 02:52:11.573261 2659 raft_consensus.cc:2781] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:52:11.573421 2659 raft_consensus.cc:482] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:52:11.573530 2659 raft_consensus.cc:3037] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:11.575995 2659 raft_consensus.cc:504] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:11.576630 2659 leader_election.cc:288] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea [CANDIDATE]: Term 1 election: Requested vote from peers 9f368a6470774802a059bea680da7b20 (127.0.97.1:44893), 8c0a4114184647fb9fa556aec7f269dd (127.0.97.2:36181)
I20260620 02:52:11.577072 2304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfcfe206467549feb7227bdb9101ab48" candidate_uuid: "45e33b6b3f894bc884f6c093deed65ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f368a6470774802a059bea680da7b20"
I20260620 02:52:11.577157 2448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfcfe206467549feb7227bdb9101ab48" candidate_uuid: "45e33b6b3f894bc884f6c093deed65ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c0a4114184647fb9fa556aec7f269dd"
I20260620 02:52:11.577263 2304 raft_consensus.cc:3037] T dfcfe206467549feb7227bdb9101ab48 P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:11.577386 2448 raft_consensus.cc:3037] T dfcfe206467549feb7227bdb9101ab48 P 8c0a4114184647fb9fa556aec7f269dd [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:11.579854 2304 raft_consensus.cc:2454] T dfcfe206467549feb7227bdb9101ab48 P 9f368a6470774802a059bea680da7b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45e33b6b3f894bc884f6c093deed65ea in term 1.
I20260620 02:52:11.579967 2448 raft_consensus.cc:2454] T dfcfe206467549feb7227bdb9101ab48 P 8c0a4114184647fb9fa556aec7f269dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45e33b6b3f894bc884f6c093deed65ea in term 1.
I20260620 02:52:11.580261 2513 leader_election.cc:302] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45e33b6b3f894bc884f6c093deed65ea, 9f368a6470774802a059bea680da7b20; no voters:
I20260620 02:52:11.580559 2659 raft_consensus.cc:2781] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:11.580953 2659 raft_consensus.cc:686] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea [term 1 LEADER]: Becoming Leader. State: Replica: 45e33b6b3f894bc884f6c093deed65ea, State: Running, Role: LEADER
I20260620 02:52:11.581542 2659 consensus_queue.cc:237] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:11.585907 2146 catalog_manager.cc:5495] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 45e33b6b3f894bc884f6c093deed65ea (127.0.97.3). New cstate: current_term: 1 leader_uuid: "45e33b6b3f894bc884f6c093deed65ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } health_report { overall_health: HEALTHY } } }
W20260620 02:52:11.638785 2355 tablet.cc:2369] T dfcfe206467549feb7227bdb9101ab48 P 9f368a6470774802a059bea680da7b20: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 02:52:11 dist-test-slave-8p8c krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:11.680281 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:11.665024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45275 (local address 127.0.0.1:57084)
0620 02:52:11.665622 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:11.665640 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:11.665764 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:11.666165 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:11.666172 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:11.666957 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:11.667200 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:11.667209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.668782 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.668786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:11.669889 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:11.669900 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.671086 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.671091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:11.671239 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:11.671920 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:11.671939 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:11.674351 (+ 2412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:11.677889 (+ 3538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:11.677899 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:11.677959 (+ 60us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:11.678460 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:11.678775 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:11.678778 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:11.678780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:11.678912 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:11.679410 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:11.679417 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:11.679590 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 02:52:11.679721 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 02:52:11.679935 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":330,"threads_started":1}
I20260620 02:52:11.680365 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:11.664944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57084 (local address 127.0.97.62:45275)
0620 02:52:11.665113 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:11.665118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:11.665732 (+ 614us) server_negotiation.cc:407] Connection header received
0620 02:52:11.665879 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:11.665884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:11.665969 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:11.666083 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:11.667413 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.668585 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.670174 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.670877 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.671003 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:11.674592 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:11.674598 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:11.674604 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:11.674638 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:11.677758 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:11.678560 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:11.678564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:11.678566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:11.678683 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:11.679012 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:11.679015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:11.679017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:11.679264 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:11.679420 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:11.679906 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 02:52:11.680084 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 02:52:11.685809 2146 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57084:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 02:52:11.686954 2146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:52:11.693698 2284 tablet_service.cc:1467] Processing CreateTablet for tablet f6c7f99b35354d73bfe722292d894ee3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=89caefa9a86c467699f85ed870793b19]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:52:11.693723 2423 tablet_service.cc:1467] Processing CreateTablet for tablet f6c7f99b35354d73bfe722292d894ee3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=89caefa9a86c467699f85ed870793b19]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:52:11.694737 2284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6c7f99b35354d73bfe722292d894ee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:11.694737 2423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6c7f99b35354d73bfe722292d894ee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:11.694303 2562 tablet_service.cc:1467] Processing CreateTablet for tablet f6c7f99b35354d73bfe722292d894ee3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=89caefa9a86c467699f85ed870793b19]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:52:11.695230 2562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6c7f99b35354d73bfe722292d894ee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:11.700959 2653 tablet_bootstrap.cc:492] T f6c7f99b35354d73bfe722292d894ee3 P 45e33b6b3f894bc884f6c093deed65ea: Bootstrap starting.
I20260620 02:52:11.701941 2651 tablet_bootstrap.cc:492] T f6c7f99b35354d73bfe722292d894ee3 P 8c0a4114184647fb9fa556aec7f269dd: Bootstrap starting.
I20260620 02:52:11.702445 2652 tablet_bootstrap.cc:492] T f6c7f99b35354d73bfe722292d894ee3 P 9f368a6470774802a059bea680da7b20: Bootstrap starting.
I20260620 02:52:11.702546 2653 tablet_bootstrap.cc:654] T f6c7f99b35354d73bfe722292d894ee3 P 45e33b6b3f894bc884f6c093deed65ea: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:11.703682 2651 tablet_bootstrap.cc:654] T f6c7f99b35354d73bfe722292d894ee3 P 8c0a4114184647fb9fa556aec7f269dd: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:11.704095 2652 tablet_bootstrap.cc:654] T f6c7f99b35354d73bfe722292d894ee3 P 9f368a6470774802a059bea680da7b20: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:11.705027 2653 tablet_bootstrap.cc:492] T f6c7f99b35354d73bfe722292d894ee3 P 45e33b6b3f894bc884f6c093deed65ea: No bootstrap required, opened a new log
I20260620 02:52:11.705238 2653 ts_tablet_manager.cc:1397] T f6c7f99b35354d73bfe722292d894ee3 P 45e33b6b3f894bc884f6c093deed65ea: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260620 02:52:11.706053 2651 tablet_bootstrap.cc:492] T f6c7f99b35354d73bfe722292d894ee3 P 8c0a4114184647fb9fa556aec7f269dd: No bootstrap required, opened a new log
I20260620 02:52:11.706198 2651 ts_tablet_manager.cc:1397] T f6c7f99b35354d73bfe722292d894ee3 P 8c0a4114184647fb9fa556aec7f269dd: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 02:52:11.706064 2653 raft_consensus.cc:348] T f6c7f99b35354d73bfe722292d894ee3 P 45e33b6b3f894bc884f6c093deed65ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:11.706301 2653 raft_consensus.cc:374] T f6c7f99b35354d73bfe722292d894ee3 P 45e33b6b3f894bc884f6c093deed65ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:11.706418 2653 raft_consensus.cc:729] T f6c7f99b35354d73bfe722292d894ee3 P 45e33b6b3f894bc884f6c093deed65ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45e33b6b3f894bc884f6c093deed65ea, State: Initialized, Role: FOLLOWER
I20260620 02:52:11.706700 2653 consensus_queue.cc:260] T f6c7f99b35354d73bfe722292d894ee3 P 45e33b6b3f894bc884f6c093deed65ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:11.707031 2651 raft_consensus.cc:348] T f6c7f99b35354d73bfe722292d894ee3 P 8c0a4114184647fb9fa556aec7f269dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:11.707273 2651 raft_consensus.cc:374] T f6c7f99b35354d73bfe722292d894ee3 P 8c0a4114184647fb9fa556aec7f269dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:11.707135 2659 tablet_replica.cc:439] T f6c7f99b35354d73bfe722292d894ee3 P 45e33b6b3f894bc884f6c093deed65ea: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } } }
I20260620 02:52:11.707356 2651 raft_consensus.cc:729] T f6c7f99b35354d73bfe722292d894ee3 P 8c0a4114184647fb9fa556aec7f269dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c0a4114184647fb9fa556aec7f269dd, State: Initialized, Role: FOLLOWER
I20260620 02:52:11.707412 2659 tablet_replica.cc:442] T f6c7f99b35354d73bfe722292d894ee3 P 45e33b6b3f894bc884f6c093deed65ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:52:11.707589 2651 consensus_queue.cc:260] T f6c7f99b35354d73bfe722292d894ee3 P 8c0a4114184647fb9fa556aec7f269dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:11.707979 2653 ts_tablet_manager.cc:1428] T f6c7f99b35354d73bfe722292d894ee3 P 45e33b6b3f894bc884f6c093deed65ea: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 02:52:11.708209 2651 ts_tablet_manager.cc:1428] T f6c7f99b35354d73bfe722292d894ee3 P 8c0a4114184647fb9fa556aec7f269dd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 02:52:11.708257 2657 tablet_replica.cc:439] T f6c7f99b35354d73bfe722292d894ee3 P 8c0a4114184647fb9fa556aec7f269dd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } } }
I20260620 02:52:11.708616 2657 tablet_replica.cc:442] T f6c7f99b35354d73bfe722292d894ee3 P 8c0a4114184647fb9fa556aec7f269dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:52:11.709391 2652 tablet_bootstrap.cc:492] T f6c7f99b35354d73bfe722292d894ee3 P 9f368a6470774802a059bea680da7b20: No bootstrap required, opened a new log
I20260620 02:52:11.709563 2652 ts_tablet_manager.cc:1397] T f6c7f99b35354d73bfe722292d894ee3 P 9f368a6470774802a059bea680da7b20: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260620 02:52:11.710256 2652 raft_consensus.cc:348] T f6c7f99b35354d73bfe722292d894ee3 P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:11.710590 2652 raft_consensus.cc:374] T f6c7f99b35354d73bfe722292d894ee3 P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:11.710683 2652 raft_consensus.cc:729] T f6c7f99b35354d73bfe722292d894ee3 P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f368a6470774802a059bea680da7b20, State: Initialized, Role: FOLLOWER
I20260620 02:52:11.710917 2652 consensus_queue.cc:260] T f6c7f99b35354d73bfe722292d894ee3 P 9f368a6470774802a059bea680da7b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:11.711581 2652 ts_tablet_manager.cc:1428] T f6c7f99b35354d73bfe722292d894ee3 P 9f368a6470774802a059bea680da7b20: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 02:52:11.711418 2658 tablet_replica.cc:439] T f6c7f99b35354d73bfe722292d894ee3 P 9f368a6470774802a059bea680da7b20: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } } }
I20260620 02:52:11.711813 2658 tablet_replica.cc:442] T f6c7f99b35354d73bfe722292d894ee3 P 9f368a6470774802a059bea680da7b20: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:52:11.737463 2658 raft_consensus.cc:482] T f6c7f99b35354d73bfe722292d894ee3 P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:52:11.737715 2658 raft_consensus.cc:504] T f6c7f99b35354d73bfe722292d894ee3 P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:11.739387 2658 leader_election.cc:288] T f6c7f99b35354d73bfe722292d894ee3 P 9f368a6470774802a059bea680da7b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c0a4114184647fb9fa556aec7f269dd (127.0.97.2:36181), 45e33b6b3f894bc884f6c093deed65ea (127.0.97.3:36599)
I20260620 02:52:11.746790 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:11.739707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60951 (local address 127.0.97.2:36181)
0620 02:52:11.739917 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:11.739924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:11.740588 (+ 664us) server_negotiation.cc:407] Connection header received
0620 02:52:11.740815 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:11.740820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:11.740929 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:11.741018 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:11.741667 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.742520 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.744683 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.745874 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.746009 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:11.746099 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:11.746347 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 02:52:11.746490 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 02:52:11.746870 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:11.739669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36181 (local address 127.0.97.1:60951)
0620 02:52:11.740462 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:11.740484 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:11.740644 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:11.741049 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:11.741053 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:11.741079 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:11.741481 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:11.741492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.742641 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.742646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:11.744471 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:11.744485 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.746062 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.746066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:11.746176 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:11.746203 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 02:52:11.746291 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 02:52:11.746437 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":43,"thread_start_us":528,"threads_started":1}
I20260620 02:52:11.747087 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:11.739923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55169 (local address 127.0.97.3:36599)
0620 02:52:11.740122 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:11.740129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:11.740785 (+ 656us) server_negotiation.cc:407] Connection header received
0620 02:52:11.741049 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:11.741055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:11.741132 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:11.741249 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:11.741850 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.742679 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.744590 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.745880 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.746003 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:11.746102 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:11.746657 (+ 555us) server_negotiation.cc:299] Negotiation successful
0620 02:52:11.746791 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 02:52:11.747349 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:11.739732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36599 (local address 127.0.97.1:55169)
0620 02:52:11.740679 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:11.740692 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:11.740875 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:11.741270 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:11.741273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:11.741292 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:11.741661 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:11.741672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.742831 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.742835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:11.744455 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:11.744467 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.746253 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.746257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:11.746467 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:11.746507 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 02:52:11.746623 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 02:52:11.746746 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"mutex_wait_us":18,"thread_start_us":420,"threads_started":1}
I20260620 02:52:11.747910 2448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c7f99b35354d73bfe722292d894ee3" candidate_uuid: "9f368a6470774802a059bea680da7b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c0a4114184647fb9fa556aec7f269dd" is_pre_election: true
I20260620 02:52:11.748193 2448 raft_consensus.cc:2454] T f6c7f99b35354d73bfe722292d894ee3 P 8c0a4114184647fb9fa556aec7f269dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f368a6470774802a059bea680da7b20 in term 0.
I20260620 02:52:11.748198 2582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c7f99b35354d73bfe722292d894ee3" candidate_uuid: "9f368a6470774802a059bea680da7b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45e33b6b3f894bc884f6c093deed65ea" is_pre_election: true
I20260620 02:52:11.748572 2582 raft_consensus.cc:2454] T f6c7f99b35354d73bfe722292d894ee3 P 45e33b6b3f894bc884f6c093deed65ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f368a6470774802a059bea680da7b20 in term 0.
I20260620 02:52:11.748651 2237 leader_election.cc:302] T f6c7f99b35354d73bfe722292d894ee3 P 9f368a6470774802a059bea680da7b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c0a4114184647fb9fa556aec7f269dd, 9f368a6470774802a059bea680da7b20; no voters:
I20260620 02:52:11.749079 2658 raft_consensus.cc:2781] T f6c7f99b35354d73bfe722292d894ee3 P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:52:11.749205 2658 raft_consensus.cc:482] T f6c7f99b35354d73bfe722292d894ee3 P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:52:11.749297 2658 raft_consensus.cc:3037] T f6c7f99b35354d73bfe722292d894ee3 P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:11.751459 2658 raft_consensus.cc:504] T f6c7f99b35354d73bfe722292d894ee3 P 9f368a6470774802a059bea680da7b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:11.752058 2658 leader_election.cc:288] T f6c7f99b35354d73bfe722292d894ee3 P 9f368a6470774802a059bea680da7b20 [CANDIDATE]: Term 1 election: Requested vote from peers 8c0a4114184647fb9fa556aec7f269dd (127.0.97.2:36181), 45e33b6b3f894bc884f6c093deed65ea (127.0.97.3:36599)
I20260620 02:52:11.752511 2582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c7f99b35354d73bfe722292d894ee3" candidate_uuid: "9f368a6470774802a059bea680da7b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45e33b6b3f894bc884f6c093deed65ea"
I20260620 02:52:11.752526 2448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c7f99b35354d73bfe722292d894ee3" candidate_uuid: "9f368a6470774802a059bea680da7b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c0a4114184647fb9fa556aec7f269dd"
I20260620 02:52:11.752699 2582 raft_consensus.cc:3037] T f6c7f99b35354d73bfe722292d894ee3 P 45e33b6b3f894bc884f6c093deed65ea [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:11.752725 2448 raft_consensus.cc:3037] T f6c7f99b35354d73bfe722292d894ee3 P 8c0a4114184647fb9fa556aec7f269dd [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:11.754707 2448 raft_consensus.cc:2454] T f6c7f99b35354d73bfe722292d894ee3 P 8c0a4114184647fb9fa556aec7f269dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f368a6470774802a059bea680da7b20 in term 1.
I20260620 02:52:11.754770 2582 raft_consensus.cc:2454] T f6c7f99b35354d73bfe722292d894ee3 P 45e33b6b3f894bc884f6c093deed65ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f368a6470774802a059bea680da7b20 in term 1.
I20260620 02:52:11.755172 2235 leader_election.cc:302] T f6c7f99b35354d73bfe722292d894ee3 P 9f368a6470774802a059bea680da7b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45e33b6b3f894bc884f6c093deed65ea, 9f368a6470774802a059bea680da7b20; no voters:
I20260620 02:52:11.755541 2658 raft_consensus.cc:2781] T f6c7f99b35354d73bfe722292d894ee3 P 9f368a6470774802a059bea680da7b20 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:11.755962 2658 raft_consensus.cc:686] T f6c7f99b35354d73bfe722292d894ee3 P 9f368a6470774802a059bea680da7b20 [term 1 LEADER]: Becoming Leader. State: Replica: 9f368a6470774802a059bea680da7b20, State: Running, Role: LEADER
I20260620 02:52:11.756420 2658 consensus_queue.cc:237] T f6c7f99b35354d73bfe722292d894ee3 P 9f368a6470774802a059bea680da7b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:11.759917 2674 tablet_replica.cc:439] T f6c7f99b35354d73bfe722292d894ee3 P 9f368a6470774802a059bea680da7b20: TxnStatusTablet state changed. Reason: New leader 9f368a6470774802a059bea680da7b20. Latest consensus state: current_term: 1 leader_uuid: "9f368a6470774802a059bea680da7b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } } }
I20260620 02:52:11.760237 2674 tablet_replica.cc:442] T f6c7f99b35354d73bfe722292d894ee3 P 9f368a6470774802a059bea680da7b20: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:52:11.760958 2676 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 02:52:11.761256 2676 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 02:52:11.761701 2146 catalog_manager.cc:5495] T f6c7f99b35354d73bfe722292d894ee3 P 9f368a6470774802a059bea680da7b20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f368a6470774802a059bea680da7b20 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "9f368a6470774802a059bea680da7b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } health_report { overall_health: UNKNOWN } } }
I20260620 02:52:11.788427 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:11.782678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44893 (local address 127.0.0.1:55284)
0620 02:52:11.782922 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:11.782940 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:11.783059 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:11.783525 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:11.783530 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:11.783543 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:52:11.783732 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:11.783740 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.784853 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.784860 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:11.785998 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:11.786009 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.786794 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.786800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:11.786971 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:11.787021 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:52:11.787868 (+ 847us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:52:11.787887 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 02:52:11.787986 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 02:52:11.788132 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 02:52:11.788440 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:11.782807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55284 (local address 127.0.97.1:44893)
0620 02:52:11.782981 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:11.782985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:11.783002 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:52:11.783265 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:11.783270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:11.783364 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:11.783505 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:52:11.783864 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.784737 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.786139 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.786612 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.786759 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:11.787228 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:52:11.787743 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:52:11.787893 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:11.788017 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 02:52:11.788102 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 02:52:11.797079 2448 raft_consensus.cc:1261] T f6c7f99b35354d73bfe722292d894ee3 P 8c0a4114184647fb9fa556aec7f269dd [term 1 FOLLOWER]: Refusing update from remote peer 9f368a6470774802a059bea680da7b20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:11.797156 2582 raft_consensus.cc:1261] T f6c7f99b35354d73bfe722292d894ee3 P 45e33b6b3f894bc884f6c093deed65ea [term 1 FOLLOWER]: Refusing update from remote peer 9f368a6470774802a059bea680da7b20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:11.798030 2674 consensus_queue.cc:1035] T f6c7f99b35354d73bfe722292d894ee3 P 9f368a6470774802a059bea680da7b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:52:11.798309 2658 consensus_queue.cc:1035] T f6c7f99b35354d73bfe722292d894ee3 P 9f368a6470774802a059bea680da7b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:52:11.805940 2657 tablet_replica.cc:439] T f6c7f99b35354d73bfe722292d894ee3 P 8c0a4114184647fb9fa556aec7f269dd: TxnStatusTablet state changed. Reason: New leader 9f368a6470774802a059bea680da7b20. Latest consensus state: current_term: 1 leader_uuid: "9f368a6470774802a059bea680da7b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } } }
I20260620 02:52:11.806245 2657 tablet_replica.cc:442] T f6c7f99b35354d73bfe722292d894ee3 P 8c0a4114184647fb9fa556aec7f269dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:52:11.810781 2659 tablet_replica.cc:439] T f6c7f99b35354d73bfe722292d894ee3 P 45e33b6b3f894bc884f6c093deed65ea: TxnStatusTablet state changed. Reason: New leader 9f368a6470774802a059bea680da7b20. Latest consensus state: current_term: 1 leader_uuid: "9f368a6470774802a059bea680da7b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } } }
I20260620 02:52:11.811031 2679 mvcc.cc:205] Tried to move back new op lower bound from 7298760424631345152 to 7298760424478601216. Current Snapshot: MvccSnapshot[applied={T|T < 7298760424631345152}]
I20260620 02:52:11.811098 2659 tablet_replica.cc:442] T f6c7f99b35354d73bfe722292d894ee3 P 45e33b6b3f894bc884f6c093deed65ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:52:11.812275 2680 mvcc.cc:205] Tried to move back new op lower bound from 7298760424631345152 to 7298760424478601216. Current Snapshot: MvccSnapshot[applied={T|T < 7298760424631345152}]
I20260620 02:52:11.812945 2657 tablet_replica.cc:439] T f6c7f99b35354d73bfe722292d894ee3 P 8c0a4114184647fb9fa556aec7f269dd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f368a6470774802a059bea680da7b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } } }
I20260620 02:52:11.813156 2657 tablet_replica.cc:442] T f6c7f99b35354d73bfe722292d894ee3 P 8c0a4114184647fb9fa556aec7f269dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:52:11.813411 2681 mvcc.cc:205] Tried to move back new op lower bound from 7298760424631345152 to 7298760424478601216. Current Snapshot: MvccSnapshot[applied={T|T < 7298760424631345152}]
I20260620 02:52:11.814895 2659 tablet_replica.cc:439] T f6c7f99b35354d73bfe722292d894ee3 P 45e33b6b3f894bc884f6c093deed65ea: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f368a6470774802a059bea680da7b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } } }
I20260620 02:52:11.815227 2659 tablet_replica.cc:442] T f6c7f99b35354d73bfe722292d894ee3 P 45e33b6b3f894bc884f6c093deed65ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:52:11.818324 2674 tablet_replica.cc:439] T f6c7f99b35354d73bfe722292d894ee3 P 9f368a6470774802a059bea680da7b20: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f368a6470774802a059bea680da7b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } } }
I20260620 02:52:11.818320 2658 tablet_replica.cc:439] T f6c7f99b35354d73bfe722292d894ee3 P 9f368a6470774802a059bea680da7b20: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f368a6470774802a059bea680da7b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } } }
I20260620 02:52:11.818635 2674 tablet_replica.cc:442] T f6c7f99b35354d73bfe722292d894ee3 P 9f368a6470774802a059bea680da7b20: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:52:11.818632 2658 tablet_replica.cc:442] T f6c7f99b35354d73bfe722292d894ee3 P 9f368a6470774802a059bea680da7b20: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:52:11.832432 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:11.825948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38836 (local address 127.0.97.3:36599)
0620 02:52:11.826129 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:11.826135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:11.826523 (+ 388us) server_negotiation.cc:407] Connection header received
0620 02:52:11.826731 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:11.826737 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:11.826837 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:11.826983 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:52:11.827349 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.828509 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.829876 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.830563 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.830687 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:11.831111 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:52:11.831789 (+ 678us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:52:11.831929 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:11.832053 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 02:52:11.832135 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 02:52:11.848582 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:11.843005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48055 (local address 127.0.97.1:44893)
0620 02:52:11.843180 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:11.843188 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:11.843710 (+ 522us) server_negotiation.cc:407] Connection header received
0620 02:52:11.843937 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:11.843941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:11.844032 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:11.844133 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:52:11.844597 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.845363 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.846833 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.847274 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.847410 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:11.847849 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:52:11.847951 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:52:11.848042 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:11.848182 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 02:52:11.848259 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 02:52:11.848678 2692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:11.842958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44893 (local address 127.0.97.3:48055)
0620 02:52:11.843604 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:11.843620 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:11.843784 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:11.844147 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:11.844152 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:11.844162 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:52:11.844443 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:11.844451 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.845527 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.845532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:11.846686 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:11.846698 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.847438 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.847444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:11.847588 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:11.847638 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:52:11.848055 (+ 417us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:52:11.848067 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 02:52:11.848161 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 02:52:11.848293 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":325,"threads_started":1}
I20260620 02:52:11.858191 2448 raft_consensus.cc:1261] T dfcfe206467549feb7227bdb9101ab48 P 8c0a4114184647fb9fa556aec7f269dd [term 1 FOLLOWER]: Refusing update from remote peer 45e33b6b3f894bc884f6c093deed65ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:11.858517 2304 raft_consensus.cc:1261] T dfcfe206467549feb7227bdb9101ab48 P 9f368a6470774802a059bea680da7b20 [term 1 FOLLOWER]: Refusing update from remote peer 45e33b6b3f894bc884f6c093deed65ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:11.858873 2661 consensus_queue.cc:1035] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:52:11.859231 2659 consensus_queue.cc:1035] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:52:11.899055 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:11.889141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54841 (local address 127.0.97.3:36599)
0620 02:52:11.889351 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:11.889358 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:11.891021 (+ 1663us) server_negotiation.cc:407] Connection header received
0620 02:52:11.891219 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:11.891225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:11.891319 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:11.891426 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:52:11.892258 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.893355 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.897085 (+ 3730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.897537 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.897652 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:11.898191 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:52:11.898290 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:52:11.898357 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:11.898673 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 02:52:11.898785 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 02:52:11.899101 2698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:11.888817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36599 (local address 127.0.97.1:54841)
0620 02:52:11.890908 (+ 2091us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:11.890928 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:11.891070 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:11.891624 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:11.891629 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:11.891642 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:52:11.892040 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:11.892053 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.893547 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.893552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:11.896751 (+ 3199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:11.896767 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:11.897734 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:11.897740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:11.897933 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:11.898016 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:52:11.898413 (+ 397us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:52:11.898425 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 02:52:11.898514 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 02:52:11.898661 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1879,"thread_start_us":1729,"threads_started":1}
I20260620 02:52:11.944171 2146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:52:11.944471 2146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:52:11.948647 2146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e40c9788c5d45278f7b8b3f8a7f7125: Sending DeleteTablet for 3 replicas of tablet dfcfe206467549feb7227bdb9101ab48
I20260620 02:52:11.949887 2284 tablet_service.cc:1514] Processing DeleteTablet for tablet dfcfe206467549feb7227bdb9101ab48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55274
I20260620 02:52:11.949889 2423 tablet_service.cc:1514] Processing DeleteTablet for tablet dfcfe206467549feb7227bdb9101ab48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49528
I20260620 02:52:11.951182 2562 tablet_service.cc:1514] Processing DeleteTablet for tablet dfcfe206467549feb7227bdb9101ab48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38826
I20260620 02:52:11.957834 2701 tablet_replica.cc:330] T dfcfe206467549feb7227bdb9101ab48 P 8c0a4114184647fb9fa556aec7f269dd: stopping tablet replica
I20260620 02:52:11.959221 2146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:52:11.959990 2146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:52:11.966497 2701 raft_consensus.cc:2229] T dfcfe206467549feb7227bdb9101ab48 P 8c0a4114184647fb9fa556aec7f269dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:52:11.966970 2701 raft_consensus.cc:2258] T dfcfe206467549feb7227bdb9101ab48 P 8c0a4114184647fb9fa556aec7f269dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:11.968600 2283 tablet_service.cc:1467] Processing CreateTablet for tablet 6fca57b77bc84dc99b6305f4f5369adf (DEFAULT_TABLE table=test-table [id=c5f7de98550b4989adc32a2c44882abb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:11.968624 2561 tablet_service.cc:1467] Processing CreateTablet for tablet 6fca57b77bc84dc99b6305f4f5369adf (DEFAULT_TABLE table=test-table [id=c5f7de98550b4989adc32a2c44882abb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:11.968739 2423 tablet_service.cc:1467] Processing CreateTablet for tablet 6fca57b77bc84dc99b6305f4f5369adf (DEFAULT_TABLE table=test-table [id=c5f7de98550b4989adc32a2c44882abb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:11.969117 2701 ts_tablet_manager.cc:1905] T dfcfe206467549feb7227bdb9101ab48 P 8c0a4114184647fb9fa556aec7f269dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:11.969163 2283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fca57b77bc84dc99b6305f4f5369adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:11.969187 2423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fca57b77bc84dc99b6305f4f5369adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:11.969204 2561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fca57b77bc84dc99b6305f4f5369adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:11.976727 2652 tablet_bootstrap.cc:492] T 6fca57b77bc84dc99b6305f4f5369adf P 9f368a6470774802a059bea680da7b20: Bootstrap starting.
I20260620 02:52:11.977221 2651 tablet_bootstrap.cc:492] T 6fca57b77bc84dc99b6305f4f5369adf P 8c0a4114184647fb9fa556aec7f269dd: Bootstrap starting.
I20260620 02:52:11.977660 2653 tablet_bootstrap.cc:492] T 6fca57b77bc84dc99b6305f4f5369adf P 45e33b6b3f894bc884f6c093deed65ea: Bootstrap starting.
I20260620 02:52:11.978616 2652 tablet_bootstrap.cc:654] T 6fca57b77bc84dc99b6305f4f5369adf P 9f368a6470774802a059bea680da7b20: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:11.978861 2651 tablet_bootstrap.cc:654] T 6fca57b77bc84dc99b6305f4f5369adf P 8c0a4114184647fb9fa556aec7f269dd: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:11.979578 2701 ts_tablet_manager.cc:1918] T dfcfe206467549feb7227bdb9101ab48 P 8c0a4114184647fb9fa556aec7f269dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:52:11.979753 2701 log.cc:1196] T dfcfe206467549feb7227bdb9101ab48 P 8c0a4114184647fb9fa556aec7f269dd: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-1/wal/wals/dfcfe206467549feb7227bdb9101ab48
I20260620 02:52:11.979849 2653 tablet_bootstrap.cc:654] T 6fca57b77bc84dc99b6305f4f5369adf P 45e33b6b3f894bc884f6c093deed65ea: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:11.980330 2701 ts_tablet_manager.cc:1939] T dfcfe206467549feb7227bdb9101ab48 P 8c0a4114184647fb9fa556aec7f269dd: Deleting consensus metadata
I20260620 02:52:11.982303 2134 catalog_manager.cc:4841] TS 8c0a4114184647fb9fa556aec7f269dd (127.0.97.2:36181): tablet dfcfe206467549feb7227bdb9101ab48 (table test-table [id=d299fc6e18f04127a0729f49915845cb]) successfully deleted
I20260620 02:52:11.991422 2703 tablet_replica.cc:330] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea: stopping tablet replica
I20260620 02:52:11.991936 2703 raft_consensus.cc:2229] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:52:11.992714 2703 raft_consensus.cc:2258] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:11.993814 2651 tablet_bootstrap.cc:492] T 6fca57b77bc84dc99b6305f4f5369adf P 8c0a4114184647fb9fa556aec7f269dd: No bootstrap required, opened a new log
I20260620 02:52:11.994001 2651 ts_tablet_manager.cc:1397] T 6fca57b77bc84dc99b6305f4f5369adf P 8c0a4114184647fb9fa556aec7f269dd: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.003s
I20260620 02:52:11.995373 2651 raft_consensus.cc:348] T 6fca57b77bc84dc99b6305f4f5369adf P 8c0a4114184647fb9fa556aec7f269dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:11.995651 2651 raft_consensus.cc:374] T 6fca57b77bc84dc99b6305f4f5369adf P 8c0a4114184647fb9fa556aec7f269dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:11.995743 2651 raft_consensus.cc:729] T 6fca57b77bc84dc99b6305f4f5369adf P 8c0a4114184647fb9fa556aec7f269dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c0a4114184647fb9fa556aec7f269dd, State: Initialized, Role: FOLLOWER
I20260620 02:52:11.996074 2651 consensus_queue.cc:260] T 6fca57b77bc84dc99b6305f4f5369adf P 8c0a4114184647fb9fa556aec7f269dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:11.997711 2651 ts_tablet_manager.cc:1428] T 6fca57b77bc84dc99b6305f4f5369adf P 8c0a4114184647fb9fa556aec7f269dd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 02:52:12.003547 2703 ts_tablet_manager.cc:1905] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:12.004314 2653 tablet_bootstrap.cc:492] T 6fca57b77bc84dc99b6305f4f5369adf P 45e33b6b3f894bc884f6c093deed65ea: No bootstrap required, opened a new log
I20260620 02:52:12.004563 2653 ts_tablet_manager.cc:1397] T 6fca57b77bc84dc99b6305f4f5369adf P 45e33b6b3f894bc884f6c093deed65ea: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.010s
I20260620 02:52:12.005606 2653 raft_consensus.cc:348] T 6fca57b77bc84dc99b6305f4f5369adf P 45e33b6b3f894bc884f6c093deed65ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:12.005962 2653 raft_consensus.cc:374] T 6fca57b77bc84dc99b6305f4f5369adf P 45e33b6b3f894bc884f6c093deed65ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:12.006105 2653 raft_consensus.cc:729] T 6fca57b77bc84dc99b6305f4f5369adf P 45e33b6b3f894bc884f6c093deed65ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45e33b6b3f894bc884f6c093deed65ea, State: Initialized, Role: FOLLOWER
I20260620 02:52:12.006454 2653 consensus_queue.cc:260] T 6fca57b77bc84dc99b6305f4f5369adf P 45e33b6b3f894bc884f6c093deed65ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:12.007287 2653 ts_tablet_manager.cc:1428] T 6fca57b77bc84dc99b6305f4f5369adf P 45e33b6b3f894bc884f6c093deed65ea: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 02:52:12.007866 2702 tablet_replica.cc:330] T dfcfe206467549feb7227bdb9101ab48 P 9f368a6470774802a059bea680da7b20: stopping tablet replica
I20260620 02:52:12.008361 2702 raft_consensus.cc:2229] T dfcfe206467549feb7227bdb9101ab48 P 9f368a6470774802a059bea680da7b20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:52:12.008852 2652 tablet_bootstrap.cc:492] T 6fca57b77bc84dc99b6305f4f5369adf P 9f368a6470774802a059bea680da7b20: No bootstrap required, opened a new log
I20260620 02:52:12.011353 2652 ts_tablet_manager.cc:1397] T 6fca57b77bc84dc99b6305f4f5369adf P 9f368a6470774802a059bea680da7b20: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.014s
I20260620 02:52:12.012208 2652 raft_consensus.cc:348] T 6fca57b77bc84dc99b6305f4f5369adf P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:12.012575 2652 raft_consensus.cc:374] T 6fca57b77bc84dc99b6305f4f5369adf P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:12.012609 2702 raft_consensus.cc:2258] T dfcfe206467549feb7227bdb9101ab48 P 9f368a6470774802a059bea680da7b20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:12.012673 2652 raft_consensus.cc:729] T 6fca57b77bc84dc99b6305f4f5369adf P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f368a6470774802a059bea680da7b20, State: Initialized, Role: FOLLOWER
I20260620 02:52:12.014531 2703 ts_tablet_manager.cc:1918] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:52:12.014717 2703 log.cc:1196] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-2/wal/wals/dfcfe206467549feb7227bdb9101ab48
I20260620 02:52:12.015290 2703 ts_tablet_manager.cc:1939] T dfcfe206467549feb7227bdb9101ab48 P 45e33b6b3f894bc884f6c093deed65ea: Deleting consensus metadata
I20260620 02:52:12.017148 2132 catalog_manager.cc:4841] TS 45e33b6b3f894bc884f6c093deed65ea (127.0.97.3:36599): tablet dfcfe206467549feb7227bdb9101ab48 (table test-table [id=d299fc6e18f04127a0729f49915845cb]) successfully deleted
I20260620 02:52:12.017769 2652 consensus_queue.cc:260] T 6fca57b77bc84dc99b6305f4f5369adf P 9f368a6470774802a059bea680da7b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:12.018587 2652 ts_tablet_manager.cc:1428] T 6fca57b77bc84dc99b6305f4f5369adf P 9f368a6470774802a059bea680da7b20: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20260620 02:52:12.019914 2702 ts_tablet_manager.cc:1905] T dfcfe206467549feb7227bdb9101ab48 P 9f368a6470774802a059bea680da7b20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:12.028779 2702 ts_tablet_manager.cc:1918] T dfcfe206467549feb7227bdb9101ab48 P 9f368a6470774802a059bea680da7b20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:52:12.029089 2702 log.cc:1196] T dfcfe206467549feb7227bdb9101ab48 P 9f368a6470774802a059bea680da7b20: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-0/wal/wals/dfcfe206467549feb7227bdb9101ab48
I20260620 02:52:12.030041 2702 ts_tablet_manager.cc:1939] T dfcfe206467549feb7227bdb9101ab48 P 9f368a6470774802a059bea680da7b20: Deleting consensus metadata
I20260620 02:52:12.032493 2131 catalog_manager.cc:4841] TS 9f368a6470774802a059bea680da7b20 (127.0.97.1:44893): tablet dfcfe206467549feb7227bdb9101ab48 (table test-table [id=d299fc6e18f04127a0729f49915845cb]) successfully deleted
I20260620 02:52:12.242950 2687 raft_consensus.cc:482] T 6fca57b77bc84dc99b6305f4f5369adf P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:52:12.243196 2687 raft_consensus.cc:504] T 6fca57b77bc84dc99b6305f4f5369adf P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:12.244026 2687 leader_election.cc:288] T 6fca57b77bc84dc99b6305f4f5369adf P 9f368a6470774802a059bea680da7b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c0a4114184647fb9fa556aec7f269dd (127.0.97.2:36181), 45e33b6b3f894bc884f6c093deed65ea (127.0.97.3:36599)
I20260620 02:52:12.244726 2448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fca57b77bc84dc99b6305f4f5369adf" candidate_uuid: "9f368a6470774802a059bea680da7b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c0a4114184647fb9fa556aec7f269dd" is_pre_election: true
I20260620 02:52:12.244846 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fca57b77bc84dc99b6305f4f5369adf" candidate_uuid: "9f368a6470774802a059bea680da7b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45e33b6b3f894bc884f6c093deed65ea" is_pre_election: true
I20260620 02:52:12.245012 2448 raft_consensus.cc:2454] T 6fca57b77bc84dc99b6305f4f5369adf P 8c0a4114184647fb9fa556aec7f269dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f368a6470774802a059bea680da7b20 in term 0.
I20260620 02:52:12.245136 2581 raft_consensus.cc:2454] T 6fca57b77bc84dc99b6305f4f5369adf P 45e33b6b3f894bc884f6c093deed65ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f368a6470774802a059bea680da7b20 in term 0.
I20260620 02:52:12.245599 2237 leader_election.cc:302] T 6fca57b77bc84dc99b6305f4f5369adf P 9f368a6470774802a059bea680da7b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 45e33b6b3f894bc884f6c093deed65ea, 8c0a4114184647fb9fa556aec7f269dd, 9f368a6470774802a059bea680da7b20; no voters:
I20260620 02:52:12.245900 2687 raft_consensus.cc:2781] T 6fca57b77bc84dc99b6305f4f5369adf P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:52:12.246084 2687 raft_consensus.cc:482] T 6fca57b77bc84dc99b6305f4f5369adf P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:52:12.246196 2687 raft_consensus.cc:3037] T 6fca57b77bc84dc99b6305f4f5369adf P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:12.249446 2687 raft_consensus.cc:504] T 6fca57b77bc84dc99b6305f4f5369adf P 9f368a6470774802a059bea680da7b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:12.250633 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fca57b77bc84dc99b6305f4f5369adf" candidate_uuid: "9f368a6470774802a059bea680da7b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45e33b6b3f894bc884f6c093deed65ea"
I20260620 02:52:12.250679 2448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fca57b77bc84dc99b6305f4f5369adf" candidate_uuid: "9f368a6470774802a059bea680da7b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c0a4114184647fb9fa556aec7f269dd"
I20260620 02:52:12.250870 2448 raft_consensus.cc:3037] T 6fca57b77bc84dc99b6305f4f5369adf P 8c0a4114184647fb9fa556aec7f269dd [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:12.250862 2581 raft_consensus.cc:3037] T 6fca57b77bc84dc99b6305f4f5369adf P 45e33b6b3f894bc884f6c093deed65ea [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:12.253731 2448 raft_consensus.cc:2454] T 6fca57b77bc84dc99b6305f4f5369adf P 8c0a4114184647fb9fa556aec7f269dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f368a6470774802a059bea680da7b20 in term 1.
I20260620 02:52:12.254098 2581 raft_consensus.cc:2454] T 6fca57b77bc84dc99b6305f4f5369adf P 45e33b6b3f894bc884f6c093deed65ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f368a6470774802a059bea680da7b20 in term 1.
I20260620 02:52:12.254216 2237 leader_election.cc:302] T 6fca57b77bc84dc99b6305f4f5369adf P 9f368a6470774802a059bea680da7b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c0a4114184647fb9fa556aec7f269dd, 9f368a6470774802a059bea680da7b20; no voters:
I20260620 02:52:12.254534 2674 raft_consensus.cc:2781] T 6fca57b77bc84dc99b6305f4f5369adf P 9f368a6470774802a059bea680da7b20 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:12.254684 2674 raft_consensus.cc:686] T 6fca57b77bc84dc99b6305f4f5369adf P 9f368a6470774802a059bea680da7b20 [term 1 LEADER]: Becoming Leader. State: Replica: 9f368a6470774802a059bea680da7b20, State: Running, Role: LEADER
I20260620 02:52:12.254841 2687 leader_election.cc:288] T 6fca57b77bc84dc99b6305f4f5369adf P 9f368a6470774802a059bea680da7b20 [CANDIDATE]: Term 1 election: Requested vote from peers 8c0a4114184647fb9fa556aec7f269dd (127.0.97.2:36181), 45e33b6b3f894bc884f6c093deed65ea (127.0.97.3:36599)
I20260620 02:52:12.254925 2674 consensus_queue.cc:237] T 6fca57b77bc84dc99b6305f4f5369adf P 9f368a6470774802a059bea680da7b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:12.258113 2147 catalog_manager.cc:5495] T 6fca57b77bc84dc99b6305f4f5369adf P 9f368a6470774802a059bea680da7b20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f368a6470774802a059bea680da7b20 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "9f368a6470774802a059bea680da7b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } health_report { overall_health: UNKNOWN } } }
I20260620 02:52:12.301404 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:12.295960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50166 (local address 127.0.97.1:44893)
0620 02:52:12.296129 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:12.296133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:12.296357 (+ 224us) server_negotiation.cc:407] Connection header received
0620 02:52:12.296538 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:12.296542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:12.296607 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:12.296674 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:52:12.297065 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:12.297752 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:12.299660 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:12.300147 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:12.300248 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:12.300719 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:52:12.300840 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:52:12.300965 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:12.301118 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 02:52:12.301175 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 02:52:12.316210 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:12.310518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36845 (local address 127.0.97.1:44893)
0620 02:52:12.310675 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:12.310680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:12.310997 (+ 317us) server_negotiation.cc:407] Connection header received
0620 02:52:12.311164 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:12.311167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:12.311224 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:12.311298 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:52:12.311717 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:12.312447 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:12.314272 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:12.314785 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:12.314876 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:12.315426 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:52:12.315559 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:52:12.315644 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:12.315793 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 02:52:12.315872 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 02:52:12.316217 2698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:12.310587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44893 (local address 127.0.97.1:36845)
0620 02:52:12.310893 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:12.310912 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:12.311053 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:12.311341 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:12.311346 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:12.311359 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:52:12.311545 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:12.311556 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:12.312585 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:12.312591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:12.314139 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:12.314154 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:12.314952 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:12.314958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:12.315153 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:12.315202 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:52:12.315669 (+ 467us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:52:12.315678 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 02:52:12.315741 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 02:52:12.315841 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179}
I20260620 02:52:12.324774 2581 raft_consensus.cc:1261] T 6fca57b77bc84dc99b6305f4f5369adf P 45e33b6b3f894bc884f6c093deed65ea [term 1 FOLLOWER]: Refusing update from remote peer 9f368a6470774802a059bea680da7b20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:12.325099 2448 raft_consensus.cc:1261] T 6fca57b77bc84dc99b6305f4f5369adf P 8c0a4114184647fb9fa556aec7f269dd [term 1 FOLLOWER]: Refusing update from remote peer 9f368a6470774802a059bea680da7b20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:12.325321 2674 consensus_queue.cc:1035] T 6fca57b77bc84dc99b6305f4f5369adf P 9f368a6470774802a059bea680da7b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:52:12.325651 2658 consensus_queue.cc:1035] T 6fca57b77bc84dc99b6305f4f5369adf P 9f368a6470774802a059bea680da7b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 02:52:12.348387 2679 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 02:52:13.946049 2146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:52:13.946306 2146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:52:13.951332 2146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e40c9788c5d45278f7b8b3f8a7f7125: Sending DeleteTablet for 3 replicas of tablet 6fca57b77bc84dc99b6305f4f5369adf
I20260620 02:52:13.952601 2423 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fca57b77bc84dc99b6305f4f5369adf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:13 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49528
I20260620 02:52:13.953197 2562 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fca57b77bc84dc99b6305f4f5369adf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:13 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38826
I20260620 02:52:13.953380 2283 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fca57b77bc84dc99b6305f4f5369adf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:13 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55274
I20260620 02:52:13.954249 2732 tablet_replica.cc:330] T 6fca57b77bc84dc99b6305f4f5369adf P 8c0a4114184647fb9fa556aec7f269dd: stopping tablet replica
I20260620 02:52:13.954355 2733 tablet_replica.cc:330] T 6fca57b77bc84dc99b6305f4f5369adf P 45e33b6b3f894bc884f6c093deed65ea: stopping tablet replica
I20260620 02:52:13.954540 2732 raft_consensus.cc:2229] T 6fca57b77bc84dc99b6305f4f5369adf P 8c0a4114184647fb9fa556aec7f269dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:52:13.954617 2733 raft_consensus.cc:2229] T 6fca57b77bc84dc99b6305f4f5369adf P 45e33b6b3f894bc884f6c093deed65ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:52:13.954847 2732 raft_consensus.cc:2258] T 6fca57b77bc84dc99b6305f4f5369adf P 8c0a4114184647fb9fa556aec7f269dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:13.954936 2733 raft_consensus.cc:2258] T 6fca57b77bc84dc99b6305f4f5369adf P 45e33b6b3f894bc884f6c093deed65ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:13.956302 2732 ts_tablet_manager.cc:1905] T 6fca57b77bc84dc99b6305f4f5369adf P 8c0a4114184647fb9fa556aec7f269dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:13.956841 2733 ts_tablet_manager.cc:1905] T 6fca57b77bc84dc99b6305f4f5369adf P 45e33b6b3f894bc884f6c093deed65ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:13.960523 2734 tablet_replica.cc:330] T 6fca57b77bc84dc99b6305f4f5369adf P 9f368a6470774802a059bea680da7b20: stopping tablet replica
I20260620 02:52:13.960733 2734 raft_consensus.cc:2229] T 6fca57b77bc84dc99b6305f4f5369adf P 9f368a6470774802a059bea680da7b20 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:52:13.961236 2734 raft_consensus.cc:2258] T 6fca57b77bc84dc99b6305f4f5369adf P 9f368a6470774802a059bea680da7b20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:13.962339 2734 ts_tablet_manager.cc:1905] T 6fca57b77bc84dc99b6305f4f5369adf P 9f368a6470774802a059bea680da7b20: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 02:52:13 dist-test-slave-8p8c krb5kdc[2102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923933, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 02:52:14 dist-test-slave-8p8c krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923933, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:14.051774 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:14.036141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50958 (local address 127.0.97.62:45275)
0620 02:52:14.036689 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:14.036693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:14.036714 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:52:14.036764 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:14.036768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:14.036823 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:14.036942 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:14.038722 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:14.039501 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:14.040825 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:14.041220 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:14.041358 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:14.045911 (+ 4553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:14.045918 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:14.045921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:14.045960 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:14.048894 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:14.049653 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:14.049659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:14.049663 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:14.049839 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:14.050224 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:14.050229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:14.050274 (+ 45us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:14.050616 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:14.050905 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:14.051194 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 02:52:14.051419 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":258,"threads_started":1}
I20260620 02:52:14.056056 2146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:52:14.056682 2146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:52:14.064270 2283 tablet_service.cc:1467] Processing CreateTablet for tablet e53dc1f5c5ec4de5a292da0b01a04446 (DEFAULT_TABLE table=test-table [id=6994edfc47c24c77937b5f4b1eeef3d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:14.064292 2423 tablet_service.cc:1467] Processing CreateTablet for tablet e53dc1f5c5ec4de5a292da0b01a04446 (DEFAULT_TABLE table=test-table [id=6994edfc47c24c77937b5f4b1eeef3d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:14.064777 2562 tablet_service.cc:1467] Processing CreateTablet for tablet e53dc1f5c5ec4de5a292da0b01a04446 (DEFAULT_TABLE table=test-table [id=6994edfc47c24c77937b5f4b1eeef3d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:14.114919 2283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e53dc1f5c5ec4de5a292da0b01a04446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:14.114993 2733 ts_tablet_manager.cc:1918] T 6fca57b77bc84dc99b6305f4f5369adf P 45e33b6b3f894bc884f6c093deed65ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 02:52:14.115037 2734 ts_tablet_manager.cc:1918] T 6fca57b77bc84dc99b6305f4f5369adf P 9f368a6470774802a059bea680da7b20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 02:52:14.115175 2734 log.cc:1196] T 6fca57b77bc84dc99b6305f4f5369adf P 9f368a6470774802a059bea680da7b20: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-0/wal/wals/6fca57b77bc84dc99b6305f4f5369adf
I20260620 02:52:14.115168 2733 log.cc:1196] T 6fca57b77bc84dc99b6305f4f5369adf P 45e33b6b3f894bc884f6c093deed65ea: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-2/wal/wals/6fca57b77bc84dc99b6305f4f5369adf
I20260620 02:52:14.115623 2733 ts_tablet_manager.cc:1939] T 6fca57b77bc84dc99b6305f4f5369adf P 45e33b6b3f894bc884f6c093deed65ea: Deleting consensus metadata
I20260620 02:52:14.115710 2734 ts_tablet_manager.cc:1939] T 6fca57b77bc84dc99b6305f4f5369adf P 9f368a6470774802a059bea680da7b20: Deleting consensus metadata
I20260620 02:52:14.116015 2562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e53dc1f5c5ec4de5a292da0b01a04446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:14.117141 2131 catalog_manager.cc:4841] TS 9f368a6470774802a059bea680da7b20 (127.0.97.1:44893): tablet 6fca57b77bc84dc99b6305f4f5369adf (table test-table [id=c5f7de98550b4989adc32a2c44882abb]) successfully deleted
I20260620 02:52:14.117638 2423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e53dc1f5c5ec4de5a292da0b01a04446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:14.122102 2132 catalog_manager.cc:4841] TS 45e33b6b3f894bc884f6c093deed65ea (127.0.97.3:36599): tablet 6fca57b77bc84dc99b6305f4f5369adf (table test-table [id=c5f7de98550b4989adc32a2c44882abb]) successfully deleted
I20260620 02:52:14.124644 2743 tablet_bootstrap.cc:492] T e53dc1f5c5ec4de5a292da0b01a04446 P 9f368a6470774802a059bea680da7b20: Bootstrap starting.
I20260620 02:52:14.124882 2744 tablet_bootstrap.cc:492] T e53dc1f5c5ec4de5a292da0b01a04446 P 45e33b6b3f894bc884f6c093deed65ea: Bootstrap starting.
I20260620 02:52:14.125092 2732 ts_tablet_manager.cc:1918] T 6fca57b77bc84dc99b6305f4f5369adf P 8c0a4114184647fb9fa556aec7f269dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 02:52:14.125526 2732 log.cc:1196] T 6fca57b77bc84dc99b6305f4f5369adf P 8c0a4114184647fb9fa556aec7f269dd: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-1/wal/wals/6fca57b77bc84dc99b6305f4f5369adf
I20260620 02:52:14.126168 2732 ts_tablet_manager.cc:1939] T 6fca57b77bc84dc99b6305f4f5369adf P 8c0a4114184647fb9fa556aec7f269dd: Deleting consensus metadata
I20260620 02:52:14.126817 2743 tablet_bootstrap.cc:654] T e53dc1f5c5ec4de5a292da0b01a04446 P 9f368a6470774802a059bea680da7b20: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:14.126901 2744 tablet_bootstrap.cc:654] T e53dc1f5c5ec4de5a292da0b01a04446 P 45e33b6b3f894bc884f6c093deed65ea: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:14.127974 2134 catalog_manager.cc:4841] TS 8c0a4114184647fb9fa556aec7f269dd (127.0.97.2:36181): tablet 6fca57b77bc84dc99b6305f4f5369adf (table test-table [id=c5f7de98550b4989adc32a2c44882abb]) successfully deleted
I20260620 02:52:14.128123 2745 tablet_bootstrap.cc:492] T e53dc1f5c5ec4de5a292da0b01a04446 P 8c0a4114184647fb9fa556aec7f269dd: Bootstrap starting.
I20260620 02:52:14.130676 2745 tablet_bootstrap.cc:654] T e53dc1f5c5ec4de5a292da0b01a04446 P 8c0a4114184647fb9fa556aec7f269dd: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:14.130775 2743 tablet_bootstrap.cc:492] T e53dc1f5c5ec4de5a292da0b01a04446 P 9f368a6470774802a059bea680da7b20: No bootstrap required, opened a new log
I20260620 02:52:14.131036 2743 ts_tablet_manager.cc:1397] T e53dc1f5c5ec4de5a292da0b01a04446 P 9f368a6470774802a059bea680da7b20: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260620 02:52:14.131299 2744 tablet_bootstrap.cc:492] T e53dc1f5c5ec4de5a292da0b01a04446 P 45e33b6b3f894bc884f6c093deed65ea: No bootstrap required, opened a new log
I20260620 02:52:14.131445 2744 ts_tablet_manager.cc:1397] T e53dc1f5c5ec4de5a292da0b01a04446 P 45e33b6b3f894bc884f6c093deed65ea: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 02:52:14.132061 2743 raft_consensus.cc:348] T e53dc1f5c5ec4de5a292da0b01a04446 P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:14.132392 2743 raft_consensus.cc:374] T e53dc1f5c5ec4de5a292da0b01a04446 P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:14.132511 2743 raft_consensus.cc:729] T e53dc1f5c5ec4de5a292da0b01a04446 P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f368a6470774802a059bea680da7b20, State: Initialized, Role: FOLLOWER
I20260620 02:52:14.132452 2744 raft_consensus.cc:348] T e53dc1f5c5ec4de5a292da0b01a04446 P 45e33b6b3f894bc884f6c093deed65ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:14.132798 2744 raft_consensus.cc:374] T e53dc1f5c5ec4de5a292da0b01a04446 P 45e33b6b3f894bc884f6c093deed65ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:14.132942 2744 raft_consensus.cc:729] T e53dc1f5c5ec4de5a292da0b01a04446 P 45e33b6b3f894bc884f6c093deed65ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45e33b6b3f894bc884f6c093deed65ea, State: Initialized, Role: FOLLOWER
I20260620 02:52:14.132836 2743 consensus_queue.cc:260] T e53dc1f5c5ec4de5a292da0b01a04446 P 9f368a6470774802a059bea680da7b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:14.133224 2744 consensus_queue.cc:260] T e53dc1f5c5ec4de5a292da0b01a04446 P 45e33b6b3f894bc884f6c093deed65ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:14.133716 2743 ts_tablet_manager.cc:1428] T e53dc1f5c5ec4de5a292da0b01a04446 P 9f368a6470774802a059bea680da7b20: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 02:52:14.139464 2745 tablet_bootstrap.cc:492] T e53dc1f5c5ec4de5a292da0b01a04446 P 8c0a4114184647fb9fa556aec7f269dd: No bootstrap required, opened a new log
I20260620 02:52:14.139645 2745 ts_tablet_manager.cc:1397] T e53dc1f5c5ec4de5a292da0b01a04446 P 8c0a4114184647fb9fa556aec7f269dd: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260620 02:52:14.140717 2745 raft_consensus.cc:348] T e53dc1f5c5ec4de5a292da0b01a04446 P 8c0a4114184647fb9fa556aec7f269dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:14.141084 2745 raft_consensus.cc:374] T e53dc1f5c5ec4de5a292da0b01a04446 P 8c0a4114184647fb9fa556aec7f269dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:14.141206 2745 raft_consensus.cc:729] T e53dc1f5c5ec4de5a292da0b01a04446 P 8c0a4114184647fb9fa556aec7f269dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c0a4114184647fb9fa556aec7f269dd, State: Initialized, Role: FOLLOWER
I20260620 02:52:14.141492 2745 consensus_queue.cc:260] T e53dc1f5c5ec4de5a292da0b01a04446 P 8c0a4114184647fb9fa556aec7f269dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:14.142479 2744 ts_tablet_manager.cc:1428] T e53dc1f5c5ec4de5a292da0b01a04446 P 45e33b6b3f894bc884f6c093deed65ea: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260620 02:52:14.142618 2745 ts_tablet_manager.cc:1428] T e53dc1f5c5ec4de5a292da0b01a04446 P 8c0a4114184647fb9fa556aec7f269dd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 02:52:14.184492 2714 raft_consensus.cc:482] T e53dc1f5c5ec4de5a292da0b01a04446 P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:52:14.184715 2714 raft_consensus.cc:504] T e53dc1f5c5ec4de5a292da0b01a04446 P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:14.185454 2714 leader_election.cc:288] T e53dc1f5c5ec4de5a292da0b01a04446 P 9f368a6470774802a059bea680da7b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c0a4114184647fb9fa556aec7f269dd (127.0.97.2:36181), 45e33b6b3f894bc884f6c093deed65ea (127.0.97.3:36599)
I20260620 02:52:14.186105 2447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53dc1f5c5ec4de5a292da0b01a04446" candidate_uuid: "9f368a6470774802a059bea680da7b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c0a4114184647fb9fa556aec7f269dd" is_pre_election: true
I20260620 02:52:14.186121 2582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53dc1f5c5ec4de5a292da0b01a04446" candidate_uuid: "9f368a6470774802a059bea680da7b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45e33b6b3f894bc884f6c093deed65ea" is_pre_election: true
I20260620 02:52:14.186350 2447 raft_consensus.cc:2454] T e53dc1f5c5ec4de5a292da0b01a04446 P 8c0a4114184647fb9fa556aec7f269dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f368a6470774802a059bea680da7b20 in term 0.
I20260620 02:52:14.186400 2582 raft_consensus.cc:2454] T e53dc1f5c5ec4de5a292da0b01a04446 P 45e33b6b3f894bc884f6c093deed65ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f368a6470774802a059bea680da7b20 in term 0.
I20260620 02:52:14.187006 2237 leader_election.cc:302] T e53dc1f5c5ec4de5a292da0b01a04446 P 9f368a6470774802a059bea680da7b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c0a4114184647fb9fa556aec7f269dd, 9f368a6470774802a059bea680da7b20; no voters:
I20260620 02:52:14.187379 2714 raft_consensus.cc:2781] T e53dc1f5c5ec4de5a292da0b01a04446 P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:52:14.187503 2714 raft_consensus.cc:482] T e53dc1f5c5ec4de5a292da0b01a04446 P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:52:14.187618 2714 raft_consensus.cc:3037] T e53dc1f5c5ec4de5a292da0b01a04446 P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:14.214561 2714 raft_consensus.cc:504] T e53dc1f5c5ec4de5a292da0b01a04446 P 9f368a6470774802a059bea680da7b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:14.215303 2714 leader_election.cc:288] T e53dc1f5c5ec4de5a292da0b01a04446 P 9f368a6470774802a059bea680da7b20 [CANDIDATE]: Term 1 election: Requested vote from peers 8c0a4114184647fb9fa556aec7f269dd (127.0.97.2:36181), 45e33b6b3f894bc884f6c093deed65ea (127.0.97.3:36599)
I20260620 02:52:14.215981 2447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53dc1f5c5ec4de5a292da0b01a04446" candidate_uuid: "9f368a6470774802a059bea680da7b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c0a4114184647fb9fa556aec7f269dd"
I20260620 02:52:14.216058 2582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53dc1f5c5ec4de5a292da0b01a04446" candidate_uuid: "9f368a6470774802a059bea680da7b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45e33b6b3f894bc884f6c093deed65ea"
I20260620 02:52:14.216228 2447 raft_consensus.cc:3037] T e53dc1f5c5ec4de5a292da0b01a04446 P 8c0a4114184647fb9fa556aec7f269dd [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:14.216300 2582 raft_consensus.cc:3037] T e53dc1f5c5ec4de5a292da0b01a04446 P 45e33b6b3f894bc884f6c093deed65ea [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:14.218302 2447 raft_consensus.cc:2454] T e53dc1f5c5ec4de5a292da0b01a04446 P 8c0a4114184647fb9fa556aec7f269dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f368a6470774802a059bea680da7b20 in term 1.
I20260620 02:52:14.218820 2237 leader_election.cc:302] T e53dc1f5c5ec4de5a292da0b01a04446 P 9f368a6470774802a059bea680da7b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c0a4114184647fb9fa556aec7f269dd, 9f368a6470774802a059bea680da7b20; no voters:
I20260620 02:52:14.219139 2714 raft_consensus.cc:2781] T e53dc1f5c5ec4de5a292da0b01a04446 P 9f368a6470774802a059bea680da7b20 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:14.219290 2714 raft_consensus.cc:686] T e53dc1f5c5ec4de5a292da0b01a04446 P 9f368a6470774802a059bea680da7b20 [term 1 LEADER]: Becoming Leader. State: Replica: 9f368a6470774802a059bea680da7b20, State: Running, Role: LEADER
I20260620 02:52:14.219595 2714 consensus_queue.cc:237] T e53dc1f5c5ec4de5a292da0b01a04446 P 9f368a6470774802a059bea680da7b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:14.219740 2582 raft_consensus.cc:2454] T e53dc1f5c5ec4de5a292da0b01a04446 P 45e33b6b3f894bc884f6c093deed65ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f368a6470774802a059bea680da7b20 in term 1.
I20260620 02:52:14.222646 2146 catalog_manager.cc:5495] T e53dc1f5c5ec4de5a292da0b01a04446 P 9f368a6470774802a059bea680da7b20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f368a6470774802a059bea680da7b20 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "9f368a6470774802a059bea680da7b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } health_report { overall_health: UNKNOWN } } }
I20260620 02:52:14.246794 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:14.241056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50174 (local address 127.0.97.1:44893)
0620 02:52:14.241623 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:14.241627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:14.241646 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:52:14.241696 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:14.241700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:14.241759 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:14.241842 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:52:14.242295 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:14.243209 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:14.244874 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:14.245412 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:14.245553 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:14.245938 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:52:14.246046 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:52:14.246114 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:14.246306 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 02:52:14.246513 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":320,"threads_started":1}
I20260620 02:52:14.255954 2447 raft_consensus.cc:1261] T e53dc1f5c5ec4de5a292da0b01a04446 P 8c0a4114184647fb9fa556aec7f269dd [term 1 FOLLOWER]: Refusing update from remote peer 9f368a6470774802a059bea680da7b20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:14.256083 2582 raft_consensus.cc:1261] T e53dc1f5c5ec4de5a292da0b01a04446 P 45e33b6b3f894bc884f6c093deed65ea [term 1 FOLLOWER]: Refusing update from remote peer 9f368a6470774802a059bea680da7b20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:14.256505 2714 consensus_queue.cc:1035] T e53dc1f5c5ec4de5a292da0b01a04446 P 9f368a6470774802a059bea680da7b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:52:14.256795 2719 consensus_queue.cc:1035] T e53dc1f5c5ec4de5a292da0b01a04446 P 9f368a6470774802a059bea680da7b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:52:14.301654 2146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:52:14.301950 2146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:52:14.305042 2146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e40c9788c5d45278f7b8b3f8a7f7125: Sending DeleteTablet for 3 replicas of tablet e53dc1f5c5ec4de5a292da0b01a04446
I20260620 02:52:14.305874 2283 tablet_service.cc:1514] Processing DeleteTablet for tablet e53dc1f5c5ec4de5a292da0b01a04446 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:14 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55274
I20260620 02:52:14.306005 2423 tablet_service.cc:1514] Processing DeleteTablet for tablet e53dc1f5c5ec4de5a292da0b01a04446 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:14 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49528
I20260620 02:52:14.306180 2734 tablet_replica.cc:330] T e53dc1f5c5ec4de5a292da0b01a04446 P 9f368a6470774802a059bea680da7b20: stopping tablet replica
I20260620 02:52:14.306334 2562 tablet_service.cc:1514] Processing DeleteTablet for tablet e53dc1f5c5ec4de5a292da0b01a04446 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:14 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38826
I20260620 02:52:14.306418 2734 raft_consensus.cc:2229] T e53dc1f5c5ec4de5a292da0b01a04446 P 9f368a6470774802a059bea680da7b20 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:52:14.306666 2733 tablet_replica.cc:330] T e53dc1f5c5ec4de5a292da0b01a04446 P 45e33b6b3f894bc884f6c093deed65ea: stopping tablet replica
I20260620 02:52:14.306674 2732 tablet_replica.cc:330] T e53dc1f5c5ec4de5a292da0b01a04446 P 8c0a4114184647fb9fa556aec7f269dd: stopping tablet replica
I20260620 02:52:14.306763 2734 raft_consensus.cc:2258] T e53dc1f5c5ec4de5a292da0b01a04446 P 9f368a6470774802a059bea680da7b20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:14.306887 2733 raft_consensus.cc:2229] T e53dc1f5c5ec4de5a292da0b01a04446 P 45e33b6b3f894bc884f6c093deed65ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:52:14.307091 2733 raft_consensus.cc:2258] T e53dc1f5c5ec4de5a292da0b01a04446 P 45e33b6b3f894bc884f6c093deed65ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:14.307337 2732 raft_consensus.cc:2229] T e53dc1f5c5ec4de5a292da0b01a04446 P 8c0a4114184647fb9fa556aec7f269dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:52:14.307216 2146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 02:52:14.307542 2732 raft_consensus.cc:2258] T e53dc1f5c5ec4de5a292da0b01a04446 P 8c0a4114184647fb9fa556aec7f269dd [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 02:52:14.307929 2146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:52:14.308418 2734 ts_tablet_manager.cc:1905] T e53dc1f5c5ec4de5a292da0b01a04446 P 9f368a6470774802a059bea680da7b20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:14.308629 2732 ts_tablet_manager.cc:1905] T e53dc1f5c5ec4de5a292da0b01a04446 P 8c0a4114184647fb9fa556aec7f269dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:14.309363 2733 ts_tablet_manager.cc:1905] T e53dc1f5c5ec4de5a292da0b01a04446 P 45e33b6b3f894bc884f6c093deed65ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:14.320005 2283 tablet_service.cc:1467] Processing CreateTablet for tablet 654a7fb703b244bb956d1d2ed21f9b91 (DEFAULT_TABLE table=test-table [id=0c4bc241a3bf490b8f4aff5eb16667df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:14.320227 2423 tablet_service.cc:1467] Processing CreateTablet for tablet 654a7fb703b244bb956d1d2ed21f9b91 (DEFAULT_TABLE table=test-table [id=0c4bc241a3bf490b8f4aff5eb16667df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:14.320257 2562 tablet_service.cc:1467] Processing CreateTablet for tablet 654a7fb703b244bb956d1d2ed21f9b91 (DEFAULT_TABLE table=test-table [id=0c4bc241a3bf490b8f4aff5eb16667df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:14.397878 2283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 654a7fb703b244bb956d1d2ed21f9b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:14.397909 2423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 654a7fb703b244bb956d1d2ed21f9b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:14.397991 2733 ts_tablet_manager.cc:1918] T e53dc1f5c5ec4de5a292da0b01a04446 P 45e33b6b3f894bc884f6c093deed65ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:52:14.398152 2733 log.cc:1196] T e53dc1f5c5ec4de5a292da0b01a04446 P 45e33b6b3f894bc884f6c093deed65ea: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-2/wal/wals/e53dc1f5c5ec4de5a292da0b01a04446
I20260620 02:52:14.398648 2733 ts_tablet_manager.cc:1939] T e53dc1f5c5ec4de5a292da0b01a04446 P 45e33b6b3f894bc884f6c093deed65ea: Deleting consensus metadata
I20260620 02:52:14.399185 2734 ts_tablet_manager.cc:1918] T e53dc1f5c5ec4de5a292da0b01a04446 P 9f368a6470774802a059bea680da7b20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:52:14.399379 2734 log.cc:1196] T e53dc1f5c5ec4de5a292da0b01a04446 P 9f368a6470774802a059bea680da7b20: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-0/wal/wals/e53dc1f5c5ec4de5a292da0b01a04446
I20260620 02:52:14.399855 2734 ts_tablet_manager.cc:1939] T e53dc1f5c5ec4de5a292da0b01a04446 P 9f368a6470774802a059bea680da7b20: Deleting consensus metadata
I20260620 02:52:14.400904 2562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 654a7fb703b244bb956d1d2ed21f9b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:14.401214 2131 catalog_manager.cc:4841] TS 9f368a6470774802a059bea680da7b20 (127.0.97.1:44893): tablet e53dc1f5c5ec4de5a292da0b01a04446 (table test-table [id=6994edfc47c24c77937b5f4b1eeef3d9]) successfully deleted
I20260620 02:52:14.402182 2732 ts_tablet_manager.cc:1918] T e53dc1f5c5ec4de5a292da0b01a04446 P 8c0a4114184647fb9fa556aec7f269dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:52:14.402451 2732 log.cc:1196] T e53dc1f5c5ec4de5a292da0b01a04446 P 8c0a4114184647fb9fa556aec7f269dd: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-1/wal/wals/e53dc1f5c5ec4de5a292da0b01a04446
I20260620 02:52:14.402989 2732 ts_tablet_manager.cc:1939] T e53dc1f5c5ec4de5a292da0b01a04446 P 8c0a4114184647fb9fa556aec7f269dd: Deleting consensus metadata
I20260620 02:52:14.404368 2132 catalog_manager.cc:4841] TS 45e33b6b3f894bc884f6c093deed65ea (127.0.97.3:36599): tablet e53dc1f5c5ec4de5a292da0b01a04446 (table test-table [id=6994edfc47c24c77937b5f4b1eeef3d9]) successfully deleted
I20260620 02:52:14.405011 2134 catalog_manager.cc:4841] TS 8c0a4114184647fb9fa556aec7f269dd (127.0.97.2:36181): tablet e53dc1f5c5ec4de5a292da0b01a04446 (table test-table [id=6994edfc47c24c77937b5f4b1eeef3d9]) successfully deleted
I20260620 02:52:14.405419 2745 tablet_bootstrap.cc:492] T 654a7fb703b244bb956d1d2ed21f9b91 P 8c0a4114184647fb9fa556aec7f269dd: Bootstrap starting.
I20260620 02:52:14.406440 2743 tablet_bootstrap.cc:492] T 654a7fb703b244bb956d1d2ed21f9b91 P 9f368a6470774802a059bea680da7b20: Bootstrap starting.
I20260620 02:52:14.407055 2745 tablet_bootstrap.cc:654] T 654a7fb703b244bb956d1d2ed21f9b91 P 8c0a4114184647fb9fa556aec7f269dd: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:14.408119 2744 tablet_bootstrap.cc:492] T 654a7fb703b244bb956d1d2ed21f9b91 P 45e33b6b3f894bc884f6c093deed65ea: Bootstrap starting.
I20260620 02:52:14.408185 2743 tablet_bootstrap.cc:654] T 654a7fb703b244bb956d1d2ed21f9b91 P 9f368a6470774802a059bea680da7b20: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:14.410179 2744 tablet_bootstrap.cc:654] T 654a7fb703b244bb956d1d2ed21f9b91 P 45e33b6b3f894bc884f6c093deed65ea: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:14.410234 2745 tablet_bootstrap.cc:492] T 654a7fb703b244bb956d1d2ed21f9b91 P 8c0a4114184647fb9fa556aec7f269dd: No bootstrap required, opened a new log
I20260620 02:52:14.410261 2743 tablet_bootstrap.cc:492] T 654a7fb703b244bb956d1d2ed21f9b91 P 9f368a6470774802a059bea680da7b20: No bootstrap required, opened a new log
I20260620 02:52:14.410415 2745 ts_tablet_manager.cc:1397] T 654a7fb703b244bb956d1d2ed21f9b91 P 8c0a4114184647fb9fa556aec7f269dd: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 02:52:14.410487 2743 ts_tablet_manager.cc:1397] T 654a7fb703b244bb956d1d2ed21f9b91 P 9f368a6470774802a059bea680da7b20: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260620 02:52:14.411352 2745 raft_consensus.cc:348] T 654a7fb703b244bb956d1d2ed21f9b91 P 8c0a4114184647fb9fa556aec7f269dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:14.411352 2743 raft_consensus.cc:348] T 654a7fb703b244bb956d1d2ed21f9b91 P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:14.411619 2745 raft_consensus.cc:374] T 654a7fb703b244bb956d1d2ed21f9b91 P 8c0a4114184647fb9fa556aec7f269dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:14.411633 2743 raft_consensus.cc:374] T 654a7fb703b244bb956d1d2ed21f9b91 P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:14.411711 2745 raft_consensus.cc:729] T 654a7fb703b244bb956d1d2ed21f9b91 P 8c0a4114184647fb9fa556aec7f269dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c0a4114184647fb9fa556aec7f269dd, State: Initialized, Role: FOLLOWER
I20260620 02:52:14.411711 2743 raft_consensus.cc:729] T 654a7fb703b244bb956d1d2ed21f9b91 P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f368a6470774802a059bea680da7b20, State: Initialized, Role: FOLLOWER
I20260620 02:52:14.411957 2743 consensus_queue.cc:260] T 654a7fb703b244bb956d1d2ed21f9b91 P 9f368a6470774802a059bea680da7b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:14.411957 2745 consensus_queue.cc:260] T 654a7fb703b244bb956d1d2ed21f9b91 P 8c0a4114184647fb9fa556aec7f269dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:14.412110 2744 tablet_bootstrap.cc:492] T 654a7fb703b244bb956d1d2ed21f9b91 P 45e33b6b3f894bc884f6c093deed65ea: No bootstrap required, opened a new log
I20260620 02:52:14.412282 2744 ts_tablet_manager.cc:1397] T 654a7fb703b244bb956d1d2ed21f9b91 P 45e33b6b3f894bc884f6c093deed65ea: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 02:52:14.412469 2745 ts_tablet_manager.cc:1428] T 654a7fb703b244bb956d1d2ed21f9b91 P 8c0a4114184647fb9fa556aec7f269dd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 02:52:14.412467 2743 ts_tablet_manager.cc:1428] T 654a7fb703b244bb956d1d2ed21f9b91 P 9f368a6470774802a059bea680da7b20: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 02:52:14.412988 2744 raft_consensus.cc:348] T 654a7fb703b244bb956d1d2ed21f9b91 P 45e33b6b3f894bc884f6c093deed65ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:14.413272 2744 raft_consensus.cc:374] T 654a7fb703b244bb956d1d2ed21f9b91 P 45e33b6b3f894bc884f6c093deed65ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:14.413364 2744 raft_consensus.cc:729] T 654a7fb703b244bb956d1d2ed21f9b91 P 45e33b6b3f894bc884f6c093deed65ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45e33b6b3f894bc884f6c093deed65ea, State: Initialized, Role: FOLLOWER
I20260620 02:52:14.413627 2744 consensus_queue.cc:260] T 654a7fb703b244bb956d1d2ed21f9b91 P 45e33b6b3f894bc884f6c093deed65ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:14.414146 2744 ts_tablet_manager.cc:1428] T 654a7fb703b244bb956d1d2ed21f9b91 P 45e33b6b3f894bc884f6c093deed65ea: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 02:52:14.554028 2750 raft_consensus.cc:482] T 654a7fb703b244bb956d1d2ed21f9b91 P 8c0a4114184647fb9fa556aec7f269dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:52:14.554258 2750 raft_consensus.cc:504] T 654a7fb703b244bb956d1d2ed21f9b91 P 8c0a4114184647fb9fa556aec7f269dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:14.555907 2750 leader_election.cc:288] T 654a7fb703b244bb956d1d2ed21f9b91 P 8c0a4114184647fb9fa556aec7f269dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f368a6470774802a059bea680da7b20 (127.0.97.1:44893), 45e33b6b3f894bc884f6c093deed65ea (127.0.97.3:36599)
I20260620 02:52:14.563122 2760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:14.556192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36599 (local address 127.0.97.2:40081)
0620 02:52:14.557570 (+ 1378us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:14.557587 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:14.557780 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:14.558127 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:14.558133 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:14.558159 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:14.558521 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:14.558531 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:14.559589 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:14.559595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:14.561085 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:14.561096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:14.562242 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:14.562248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:14.562425 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:14.562468 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 02:52:14.562612 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 02:52:14.562738 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1267,"mutex_wait_us":57,"thread_start_us":1007,"threads_started":1}
I20260620 02:52:14.563145 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:14.556307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40081 (local address 127.0.97.3:36599)
0620 02:52:14.557266 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:14.557275 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:14.557687 (+ 412us) server_negotiation.cc:407] Connection header received
0620 02:52:14.557922 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:14.557931 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:14.557994 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:14.558098 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:14.558684 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:14.559444 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:14.561231 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:14.562027 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:14.562176 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:14.562325 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:14.562662 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 02:52:14.562816 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":799,"thread_start_us":639,"threads_started":1}
I20260620 02:52:14.563194 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:14.556529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56195 (local address 127.0.97.1:44893)
0620 02:52:14.556770 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:14.556780 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:14.558035 (+ 1255us) server_negotiation.cc:407] Connection header received
0620 02:52:14.558223 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:14.558234 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:14.558294 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:14.558416 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:14.559044 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:14.559720 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:14.561448 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:14.562213 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:14.562376 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:14.562516 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:14.562848 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 02:52:14.562980 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 02:52:14.564123 2582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654a7fb703b244bb956d1d2ed21f9b91" candidate_uuid: "8c0a4114184647fb9fa556aec7f269dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45e33b6b3f894bc884f6c093deed65ea" is_pre_election: true
I20260620 02:52:14.564375 2582 raft_consensus.cc:2454] T 654a7fb703b244bb956d1d2ed21f9b91 P 45e33b6b3f894bc884f6c093deed65ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c0a4114184647fb9fa556aec7f269dd in term 0.
I20260620 02:52:14.564431 2762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:14.556340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44893 (local address 127.0.97.2:56195)
0620 02:52:14.557909 (+ 1569us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:14.557923 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:14.558076 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:14.558444 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:14.558448 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:14.558474 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:14.558827 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:14.558835 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:14.559868 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:14.559876 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:14.561305 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:14.561316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:14.562460 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:14.562466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:14.562673 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:14.562703 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 02:52:14.563338 (+ 635us) client_negotiation.cc:240] Negotiation successful
0620 02:52:14.563537 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1464,"mutex_wait_us":4,"thread_start_us":1249,"threads_started":1}
I20260620 02:52:14.564844 2375 leader_election.cc:302] T 654a7fb703b244bb956d1d2ed21f9b91 P 8c0a4114184647fb9fa556aec7f269dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45e33b6b3f894bc884f6c093deed65ea, 8c0a4114184647fb9fa556aec7f269dd; no voters:
I20260620 02:52:14.565088 2304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654a7fb703b244bb956d1d2ed21f9b91" candidate_uuid: "8c0a4114184647fb9fa556aec7f269dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f368a6470774802a059bea680da7b20" is_pre_election: true
I20260620 02:52:14.565255 2750 raft_consensus.cc:2781] T 654a7fb703b244bb956d1d2ed21f9b91 P 8c0a4114184647fb9fa556aec7f269dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:52:14.565323 2304 raft_consensus.cc:2454] T 654a7fb703b244bb956d1d2ed21f9b91 P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c0a4114184647fb9fa556aec7f269dd in term 0.
I20260620 02:52:14.565382 2750 raft_consensus.cc:482] T 654a7fb703b244bb956d1d2ed21f9b91 P 8c0a4114184647fb9fa556aec7f269dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:52:14.565462 2750 raft_consensus.cc:3037] T 654a7fb703b244bb956d1d2ed21f9b91 P 8c0a4114184647fb9fa556aec7f269dd [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:14.567713 2750 raft_consensus.cc:504] T 654a7fb703b244bb956d1d2ed21f9b91 P 8c0a4114184647fb9fa556aec7f269dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:14.568296 2750 leader_election.cc:288] T 654a7fb703b244bb956d1d2ed21f9b91 P 8c0a4114184647fb9fa556aec7f269dd [CANDIDATE]: Term 1 election: Requested vote from peers 9f368a6470774802a059bea680da7b20 (127.0.97.1:44893), 45e33b6b3f894bc884f6c093deed65ea (127.0.97.3:36599)
I20260620 02:52:14.568743 2582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654a7fb703b244bb956d1d2ed21f9b91" candidate_uuid: "8c0a4114184647fb9fa556aec7f269dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45e33b6b3f894bc884f6c093deed65ea"
I20260620 02:52:14.568737 2304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654a7fb703b244bb956d1d2ed21f9b91" candidate_uuid: "8c0a4114184647fb9fa556aec7f269dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f368a6470774802a059bea680da7b20"
I20260620 02:52:14.568950 2582 raft_consensus.cc:3037] T 654a7fb703b244bb956d1d2ed21f9b91 P 45e33b6b3f894bc884f6c093deed65ea [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:14.568982 2304 raft_consensus.cc:3037] T 654a7fb703b244bb956d1d2ed21f9b91 P 9f368a6470774802a059bea680da7b20 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:14.571040 2582 raft_consensus.cc:2454] T 654a7fb703b244bb956d1d2ed21f9b91 P 45e33b6b3f894bc884f6c093deed65ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c0a4114184647fb9fa556aec7f269dd in term 1.
I20260620 02:52:14.571218 2304 raft_consensus.cc:2454] T 654a7fb703b244bb956d1d2ed21f9b91 P 9f368a6470774802a059bea680da7b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c0a4114184647fb9fa556aec7f269dd in term 1.
I20260620 02:52:14.571416 2375 leader_election.cc:302] T 654a7fb703b244bb956d1d2ed21f9b91 P 8c0a4114184647fb9fa556aec7f269dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45e33b6b3f894bc884f6c093deed65ea, 8c0a4114184647fb9fa556aec7f269dd; no voters:
I20260620 02:52:14.571704 2750 raft_consensus.cc:2781] T 654a7fb703b244bb956d1d2ed21f9b91 P 8c0a4114184647fb9fa556aec7f269dd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:14.572191 2750 raft_consensus.cc:686] T 654a7fb703b244bb956d1d2ed21f9b91 P 8c0a4114184647fb9fa556aec7f269dd [term 1 LEADER]: Becoming Leader. State: Replica: 8c0a4114184647fb9fa556aec7f269dd, State: Running, Role: LEADER
I20260620 02:52:14.572674 2750 consensus_queue.cc:237] T 654a7fb703b244bb956d1d2ed21f9b91 P 8c0a4114184647fb9fa556aec7f269dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } }
I20260620 02:52:14.576203 2146 catalog_manager.cc:5495] T 654a7fb703b244bb956d1d2ed21f9b91 P 8c0a4114184647fb9fa556aec7f269dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c0a4114184647fb9fa556aec7f269dd (127.0.97.2). New cstate: current_term: 1 leader_uuid: "8c0a4114184647fb9fa556aec7f269dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c0a4114184647fb9fa556aec7f269dd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 } health_report { overall_health: UNKNOWN } } }
I20260620 02:52:14.640851 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:14.634354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36620 (local address 127.0.97.2:36181)
0620 02:52:14.635351 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:14.635356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:14.635375 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:52:14.635471 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:14.635474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:14.635547 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:14.635689 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:52:14.636096 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:14.636869 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:14.638362 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:14.638885 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:14.638982 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:14.639468 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:52:14.640102 (+ 634us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:52:14.640230 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:14.640390 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 02:52:14.640465 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":882,"thread_start_us":767,"threads_started":1}
I20260620 02:52:14.657238 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:14.651076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35925 (local address 127.0.97.1:44893)
0620 02:52:14.651269 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:14.651274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:14.651860 (+ 586us) server_negotiation.cc:407] Connection header received
0620 02:52:14.652110 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:14.652114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:14.652181 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:14.652284 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:52:14.652741 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:14.653499 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:14.655498 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:14.655911 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:14.656060 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:14.656491 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:52:14.656594 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:52:14.656689 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:14.656825 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 02:52:14.656907 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 02:52:14.657348 2768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:14.651091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44893 (local address 127.0.97.2:35925)
0620 02:52:14.651747 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:14.651767 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:14.651938 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:14.652297 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:14.652302 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:14.652312 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:52:14.652592 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:14.652599 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:14.653655 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:14.653660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:14.655358 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:14.655369 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:14.656088 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:14.656094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:14.656245 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:14.656303 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:52:14.656689 (+ 386us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:52:14.656702 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 02:52:14.656805 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 02:52:14.656940 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":325,"threads_started":1}
I20260620 02:52:14.667889 2582 raft_consensus.cc:1261] T 654a7fb703b244bb956d1d2ed21f9b91 P 45e33b6b3f894bc884f6c093deed65ea [term 1 FOLLOWER]: Refusing update from remote peer 8c0a4114184647fb9fa556aec7f269dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:14.668081 2304 raft_consensus.cc:1261] T 654a7fb703b244bb956d1d2ed21f9b91 P 9f368a6470774802a059bea680da7b20 [term 1 FOLLOWER]: Refusing update from remote peer 8c0a4114184647fb9fa556aec7f269dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:14.668663 2763 consensus_queue.cc:1035] T 654a7fb703b244bb956d1d2ed21f9b91 P 8c0a4114184647fb9fa556aec7f269dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "45e33b6b3f894bc884f6c093deed65ea" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:52:14.669301 2763 consensus_queue.cc:1035] T 654a7fb703b244bb956d1d2ed21f9b91 P 8c0a4114184647fb9fa556aec7f269dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f368a6470774802a059bea680da7b20" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:52:14.707046 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:14.700677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52459 (local address 127.0.97.2:36181)
0620 02:52:14.701153 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:14.701159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:14.701518 (+ 359us) server_negotiation.cc:407] Connection header received
0620 02:52:14.701699 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:14.701703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:14.701766 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:14.701840 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:52:14.702542 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:14.703401 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:14.705352 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:14.705765 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:14.705908 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:14.706301 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:52:14.706462 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:52:14.706558 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:14.706704 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 02:52:14.706788 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352}
I20260620 02:52:14.707170 2774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:14.700716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36181 (local address 127.0.97.1:52459)
0620 02:52:14.701410 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:14.701435 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:14.701563 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:14.701876 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:14.701881 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:14.701893 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:52:14.702408 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:14.702421 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:14.703538 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:14.703544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:14.705200 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:14.705212 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:14.705942 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:14.705948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:14.706104 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:14.706139 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:52:14.706590 (+ 451us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:52:14.706600 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 02:52:14.706674 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 02:52:14.706806 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":457,"threads_started":1}
W20260620 02:52:14.708585 2716 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 02:52:16.322244 2146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:52:16.322489 2146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:52:16.326740 2146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e40c9788c5d45278f7b8b3f8a7f7125: Sending DeleteTablet for 3 replicas of tablet 654a7fb703b244bb956d1d2ed21f9b91
I20260620 02:52:16.327615 2283 tablet_service.cc:1514] Processing DeleteTablet for tablet 654a7fb703b244bb956d1d2ed21f9b91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55274
I20260620 02:52:16.327623 2423 tablet_service.cc:1514] Processing DeleteTablet for tablet 654a7fb703b244bb956d1d2ed21f9b91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49528
I20260620 02:52:16.327680 2562 tablet_service.cc:1514] Processing DeleteTablet for tablet 654a7fb703b244bb956d1d2ed21f9b91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38826
I20260620 02:52:16.328542 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 2218
I20260620 02:52:16.340358 2798 tablet_replica.cc:330] T 654a7fb703b244bb956d1d2ed21f9b91 P 9f368a6470774802a059bea680da7b20: stopping tablet replica
I20260620 02:52:16.340814 2798 raft_consensus.cc:2229] T 654a7fb703b244bb956d1d2ed21f9b91 P 9f368a6470774802a059bea680da7b20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:52:16.341212 2798 raft_consensus.cc:2258] T 654a7fb703b244bb956d1d2ed21f9b91 P 9f368a6470774802a059bea680da7b20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:16.342592 2798 ts_tablet_manager.cc:1905] T 654a7fb703b244bb956d1d2ed21f9b91 P 9f368a6470774802a059bea680da7b20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:16.344882 2799 tablet_replica.cc:330] T 654a7fb703b244bb956d1d2ed21f9b91 P 45e33b6b3f894bc884f6c093deed65ea: stopping tablet replica
I20260620 02:52:16.345130 2797 tablet_replica.cc:330] T 654a7fb703b244bb956d1d2ed21f9b91 P 8c0a4114184647fb9fa556aec7f269dd: stopping tablet replica
I20260620 02:52:16.345155 2799 raft_consensus.cc:2229] T 654a7fb703b244bb956d1d2ed21f9b91 P 45e33b6b3f894bc884f6c093deed65ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:52:16.345394 2799 raft_consensus.cc:2258] T 654a7fb703b244bb956d1d2ed21f9b91 P 45e33b6b3f894bc884f6c093deed65ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:16.345468 2797 raft_consensus.cc:2229] T 654a7fb703b244bb956d1d2ed21f9b91 P 8c0a4114184647fb9fa556aec7f269dd [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:52:16.346065 2797 raft_consensus.cc:2258] T 654a7fb703b244bb956d1d2ed21f9b91 P 8c0a4114184647fb9fa556aec7f269dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:16.346935 2799 ts_tablet_manager.cc:1905] T 654a7fb703b244bb956d1d2ed21f9b91 P 45e33b6b3f894bc884f6c093deed65ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:16.347088 2797 ts_tablet_manager.cc:1905] T 654a7fb703b244bb956d1d2ed21f9b91 P 8c0a4114184647fb9fa556aec7f269dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:16.350699 2798 ts_tablet_manager.cc:1918] T 654a7fb703b244bb956d1d2ed21f9b91 P 9f368a6470774802a059bea680da7b20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 02:52:16.350867 2798 log.cc:1196] T 654a7fb703b244bb956d1d2ed21f9b91 P 9f368a6470774802a059bea680da7b20: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-0/wal/wals/654a7fb703b244bb956d1d2ed21f9b91
I20260620 02:52:16.351470 2798 ts_tablet_manager.cc:1939] T 654a7fb703b244bb956d1d2ed21f9b91 P 9f368a6470774802a059bea680da7b20: Deleting consensus metadata
I20260620 02:52:16.352749 2131 catalog_manager.cc:4841] TS 9f368a6470774802a059bea680da7b20 (127.0.97.1:44893): tablet 654a7fb703b244bb956d1d2ed21f9b91 (table test-table [id=0c4bc241a3bf490b8f4aff5eb16667df]) successfully deleted
I20260620 02:52:16.353808 2797 ts_tablet_manager.cc:1918] T 654a7fb703b244bb956d1d2ed21f9b91 P 8c0a4114184647fb9fa556aec7f269dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 02:52:16.353822 2799 ts_tablet_manager.cc:1918] T 654a7fb703b244bb956d1d2ed21f9b91 P 45e33b6b3f894bc884f6c093deed65ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 02:52:16.353966 2797 log.cc:1196] T 654a7fb703b244bb956d1d2ed21f9b91 P 8c0a4114184647fb9fa556aec7f269dd: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-1/wal/wals/654a7fb703b244bb956d1d2ed21f9b91
I20260620 02:52:16.353971 2799 log.cc:1196] T 654a7fb703b244bb956d1d2ed21f9b91 P 45e33b6b3f894bc884f6c093deed65ea: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781923914418765-388-0/minicluster-data/ts-2/wal/wals/654a7fb703b244bb956d1d2ed21f9b91
I20260620 02:52:16.354420 2799 ts_tablet_manager.cc:1939] T 654a7fb703b244bb956d1d2ed21f9b91 P 45e33b6b3f894bc884f6c093deed65ea: Deleting consensus metadata
I20260620 02:52:16.354638 2797 ts_tablet_manager.cc:1939] T 654a7fb703b244bb956d1d2ed21f9b91 P 8c0a4114184647fb9fa556aec7f269dd: Deleting consensus metadata
I20260620 02:52:16.355624 2132 catalog_manager.cc:4841] TS 45e33b6b3f894bc884f6c093deed65ea (127.0.97.3:36599): tablet 654a7fb703b244bb956d1d2ed21f9b91 (table test-table [id=0c4bc241a3bf490b8f4aff5eb16667df]) successfully deleted
I20260620 02:52:16.355765 2134 catalog_manager.cc:4841] TS 8c0a4114184647fb9fa556aec7f269dd (127.0.97.2:36181): tablet 654a7fb703b244bb956d1d2ed21f9b91 (table test-table [id=0c4bc241a3bf490b8f4aff5eb16667df]) successfully deleted
Jun 20 02:52:16 dist-test-slave-8p8c krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923933, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:52:16.364341 2800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:16.331253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50182 (local address 127.0.97.1:44893)
0620 02:52:16.340288 (+ 9035us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:16.340297 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:16.342696 (+ 2399us) server_negotiation.cc:407] Connection header received
0620 02:52:16.342925 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:16.342943 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:16.343020 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:16.343126 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:16.345080 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:16.346354 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:16.348622 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:16.349389 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:16.349514 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:16.358705 (+ 9191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:16.358723 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:16.358740 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:16.358781 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:16.361668 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:16.362356 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:16.362409 (+ 53us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:16.362413 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:16.362606 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:16.363005 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:16.363009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:16.363011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:16.363349 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:16.363694 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:16.363902 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 02:52:16.364079 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8833,"thread_start_us":8651,"threads_started":1}
W20260620 02:52:16.365041 2349 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50182
W20260620 02:52:16.365713 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 2218: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:52:16.365964 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 2218
I20260620 02:52:16.391772 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 2358
Jun 20 02:52:16 dist-test-slave-8p8c krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923933, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:52:16.407359 2802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:16.392576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36622 (local address 127.0.97.2:36181)
0620 02:52:16.393094 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:16.393098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:16.393113 (+ 15us) server_negotiation.cc:407] Connection header received
0620 02:52:16.393165 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:16.393169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:16.393231 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:16.393320 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:16.394902 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:16.395621 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:16.396933 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:16.397402 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:16.397518 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:16.401681 (+ 4163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:16.401693 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:16.401709 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:16.401741 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:16.404579 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:16.405259 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:16.405266 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:16.405268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:16.405399 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:16.405739 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:16.405742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:16.405744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:16.406070 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:16.406275 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:16.406892 (+ 617us) server_negotiation.cc:299] Negotiation successful
0620 02:52:16.407098 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":288,"threads_started":1}
W20260620 02:52:16.408046 2488 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36622
W20260620 02:52:16.408804 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 2358: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:52:16.409343 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 2358
I20260620 02:52:16.433799 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 2497
Jun 20 02:52:16 dist-test-slave-8p8c krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923933, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:52:16.450299 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:16.434968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43242 (local address 127.0.97.3:36599)
0620 02:52:16.435587 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:16.435596 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:16.435707 (+ 111us) server_negotiation.cc:407] Connection header received
0620 02:52:16.435896 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:16.435900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:16.436002 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:16.436128 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:16.437578 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:16.438331 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:16.439656 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:16.440034 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:16.440156 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:16.444436 (+ 4280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:16.444450 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:16.444466 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:16.444496 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:16.447753 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:16.448424 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:16.448432 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:16.448434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:16.448572 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:16.448942 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:16.448946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:16.448948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:16.449299 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:16.449540 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:16.449868 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 02:52:16.450032 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":314,"threads_started":1}
W20260620 02:52:16.451114 2626 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43242
W20260620 02:52:16.451790 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 2497: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:52:16.452011 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 2497
I20260620 02:52:16.475593 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 2115
I20260620 02:52:16.476843 2187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:16.591926 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 2115
2026-06-20T02:52:16Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8831 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:52:16 dist-test-slave-8p8c krb5kdc[2807](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:52:16 dist-test-slave-8p8c krb5kdc[2807](info): set up 2 sockets
Jun 20 02:52:16 dist-test-slave-8p8c krb5kdc[2807](info): commencing operation
krb5kdc: starting...
W20260620 02:52:17.698784 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:52:17 dist-test-slave-8p8c krb5kdc[2807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:52:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:52:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:52:17.895866 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34209
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34209
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:18.220911 2820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:18.221225 2820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:18.221391 2820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:18.228742 2820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:52:18.228863 2820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:18.228950 2820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:18.229067 2820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:52:18.229133 2820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:52:18.239887 2820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:18.241304 2820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:18.242753 2820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:18.255642 2825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:18.255729 2826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:18.255769 2820 server_base.cc:1034] running on GCE node
W20260620 02:52:18.255703 2828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:18.256878 2820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:18.258251 2820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:18.259452 2820 hybrid_clock.cc:648] HybridClock initialized: now 1781923938259411 us; error 25 us; skew 500 ppm
I20260620 02:52:18.259859 2820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:18 dist-test-slave-8p8c krb5kdc[2807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:18.263002 2820 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:52:18.264458 2820 webserver.cc:460] Webserver started at http://127.0.97.62:39031/ using document root <none> and password file <none>
I20260620 02:52:18.265157 2820 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:18.265300 2820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:18.265621 2820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:18.268565 2820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "f8dd9afa64294ef2a775ac57dad94e1e"
format_stamp: "Formatted at 2026-06-20 02:52:18 on dist-test-slave-8p8c"
server_key: "9b4ceb2d7cc979d31380b77df0cc0dae"
server_key_iv: "505d77caa49062263d16ea7c98aef1ee"
server_key_version: "encryptionkey@0"
I20260620 02:52:18.269363 2820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "f8dd9afa64294ef2a775ac57dad94e1e"
format_stamp: "Formatted at 2026-06-20 02:52:18 on dist-test-slave-8p8c"
server_key: "9b4ceb2d7cc979d31380b77df0cc0dae"
server_key_iv: "505d77caa49062263d16ea7c98aef1ee"
server_key_version: "encryptionkey@0"
I20260620 02:52:18.274524 2820 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 02:52:18.277988 2835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:18.279392 2820 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 02:52:18.279587 2820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "f8dd9afa64294ef2a775ac57dad94e1e"
format_stamp: "Formatted at 2026-06-20 02:52:18 on dist-test-slave-8p8c"
server_key: "9b4ceb2d7cc979d31380b77df0cc0dae"
server_key_iv: "505d77caa49062263d16ea7c98aef1ee"
server_key_version: "encryptionkey@0"
I20260620 02:52:18.279795 2820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:18.307188 2820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:18.312068 2820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:18.312463 2820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:18.331000 2820 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34209
I20260620 02:52:18.331032 2886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34209 every 8 connection(s)
I20260620 02:52:18.332393 2820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:52:18.334818 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 2820
I20260620 02:52:18.335045 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:52:18.335654 388 external_mini_cluster.cc:1428] Setting key b166c10756e353f939aa9d57dae62784
I20260620 02:52:18.336414 2887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:18.348186 2887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8dd9afa64294ef2a775ac57dad94e1e: Bootstrap starting.
Jun 20 02:52:18 dist-test-slave-8p8c krb5kdc[2807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:18.351322 2887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8dd9afa64294ef2a775ac57dad94e1e: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:18.352334 2887 log.cc:824] T 00000000000000000000000000000000 P f8dd9afa64294ef2a775ac57dad94e1e: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:18.355072 2887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8dd9afa64294ef2a775ac57dad94e1e: No bootstrap required, opened a new log
I20260620 02:52:18.356509 2890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:18.337630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36456 (local address 127.0.97.62:34209)
0620 02:52:18.338339 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:18.338351 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:18.338442 (+ 91us) server_negotiation.cc:407] Connection header received
0620 02:52:18.339994 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:18.340014 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:18.340383 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:18.340768 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:18.342167 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:18.343507 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:18.344721 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:18.345384 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:18.345585 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:18.349613 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:18.349620 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:18.349637 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:18.349674 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:18.352741 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:18.353387 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:18.353393 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:18.353398 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:18.353553 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:18.353926 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:18.353932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:18.353935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:18.354482 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:18.354755 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:18.355207 (+ 452us) server_negotiation.cc:299] Negotiation successful
0620 02:52:18.355602 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":323,"threads_started":1}
I20260620 02:52:18.362624 2887 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8dd9afa64294ef2a775ac57dad94e1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8dd9afa64294ef2a775ac57dad94e1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34209 } }
I20260620 02:52:18.362973 2887 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8dd9afa64294ef2a775ac57dad94e1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:18.363082 2887 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8dd9afa64294ef2a775ac57dad94e1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8dd9afa64294ef2a775ac57dad94e1e, State: Initialized, Role: FOLLOWER
I20260620 02:52:18.363667 2887 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8dd9afa64294ef2a775ac57dad94e1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8dd9afa64294ef2a775ac57dad94e1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34209 } }
I20260620 02:52:18.363897 2887 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8dd9afa64294ef2a775ac57dad94e1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:52:18.364030 2887 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8dd9afa64294ef2a775ac57dad94e1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:52:18.364209 2887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8dd9afa64294ef2a775ac57dad94e1e [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:18.366526 2887 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8dd9afa64294ef2a775ac57dad94e1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8dd9afa64294ef2a775ac57dad94e1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34209 } }
I20260620 02:52:18.367018 2887 leader_election.cc:302] T 00000000000000000000000000000000 P f8dd9afa64294ef2a775ac57dad94e1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8dd9afa64294ef2a775ac57dad94e1e; no voters:
I20260620 02:52:18.367551 2887 leader_election.cc:288] T 00000000000000000000000000000000 P f8dd9afa64294ef2a775ac57dad94e1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:52:18.367704 2892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8dd9afa64294ef2a775ac57dad94e1e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:18.368384 2892 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8dd9afa64294ef2a775ac57dad94e1e [term 1 LEADER]: Becoming Leader. State: Replica: f8dd9afa64294ef2a775ac57dad94e1e, State: Running, Role: LEADER
I20260620 02:52:18.368958 2892 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8dd9afa64294ef2a775ac57dad94e1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8dd9afa64294ef2a775ac57dad94e1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34209 } }
I20260620 02:52:18.369565 2887 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8dd9afa64294ef2a775ac57dad94e1e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:52:18.371955 2894 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8dd9afa64294ef2a775ac57dad94e1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8dd9afa64294ef2a775ac57dad94e1e. Latest consensus state: current_term: 1 leader_uuid: "f8dd9afa64294ef2a775ac57dad94e1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8dd9afa64294ef2a775ac57dad94e1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34209 } } }
I20260620 02:52:18.371907 2893 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8dd9afa64294ef2a775ac57dad94e1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8dd9afa64294ef2a775ac57dad94e1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8dd9afa64294ef2a775ac57dad94e1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34209 } } }
I20260620 02:52:18.372368 2894 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8dd9afa64294ef2a775ac57dad94e1e [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:18.372417 2893 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8dd9afa64294ef2a775ac57dad94e1e [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:18.373297 2899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:52:18.379040 2899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:52:18.387735 2899 catalog_manager.cc:1347] Generated new cluster ID: fc119208c037406bb9ba76ae9fdd9f7a
I20260620 02:52:18.387856 2899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:52:18.401469 2899 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:52:18.402860 2899 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:52:18.412663 2899 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8dd9afa64294ef2a775ac57dad94e1e: Generated new TSK 0
I20260620 02:52:18.413501 2899 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:52:18.466599 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34209
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:18.789448 2913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:18.789767 2913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:18.789942 2913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:18.797133 2913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:18.797286 2913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:18.797465 2913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:52:18.807670 2913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:18.808947 2913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:18.810251 2913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:18.823355 2919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:18.823385 2921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:18.823323 2918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:18.823804 2913 server_base.cc:1034] running on GCE node
I20260620 02:52:18.824536 2913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:18.825519 2913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:18.826772 2913 hybrid_clock.cc:648] HybridClock initialized: now 1781923938826701 us; error 64 us; skew 500 ppm
I20260620 02:52:18.827118 2913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:18 dist-test-slave-8p8c krb5kdc[2807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:18.830399 2913 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:52:18.831857 2913 webserver.cc:460] Webserver started at http://127.0.97.1:40011/ using document root <none> and password file <none>
I20260620 02:52:18.832566 2913 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:18.832711 2913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:18.833061 2913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:18.836066 2913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/data/instance:
uuid: "abf40708e4a644118ae1151cf0a21b24"
format_stamp: "Formatted at 2026-06-20 02:52:18 on dist-test-slave-8p8c"
server_key: "949f4cd069453d752dd0402c1ad110b3"
server_key_iv: "b178fbf308ab1e7964fa642bc9df2edf"
server_key_version: "encryptionkey@0"
I20260620 02:52:18.836907 2913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/wal/instance:
uuid: "abf40708e4a644118ae1151cf0a21b24"
format_stamp: "Formatted at 2026-06-20 02:52:18 on dist-test-slave-8p8c"
server_key: "949f4cd069453d752dd0402c1ad110b3"
server_key_iv: "b178fbf308ab1e7964fa642bc9df2edf"
server_key_version: "encryptionkey@0"
I20260620 02:52:18.842310 2913 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 02:52:18.845969 2928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:18.847535 2913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:52:18.847718 2913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/wal
uuid: "abf40708e4a644118ae1151cf0a21b24"
format_stamp: "Formatted at 2026-06-20 02:52:18 on dist-test-slave-8p8c"
server_key: "949f4cd069453d752dd0402c1ad110b3"
server_key_iv: "b178fbf308ab1e7964fa642bc9df2edf"
server_key_version: "encryptionkey@0"
I20260620 02:52:18.847944 2913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:18.872864 2913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:18.877759 2913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:18.878105 2913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:18.879191 2913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:52:18.880571 2913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:18.880671 2913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:18.880817 2913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:18.880888 2913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:18.915535 2913 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39877
I20260620 02:52:18.915674 3040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39877 every 8 connection(s)
I20260620 02:52:18.916909 2913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:52:18.926836 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 2913
I20260620 02:52:18.927064 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:52:18.927662 388 external_mini_cluster.cc:1428] Setting key beb566fa436f175f07fa6a0630fb3a99
Jun 20 02:52:18 dist-test-slave-8p8c krb5kdc[2807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:18.938447 3043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:18.919659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55367 (local address 127.0.97.62:34209)
0620 02:52:18.920142 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:18.920146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:18.920931 (+ 785us) server_negotiation.cc:407] Connection header received
0620 02:52:18.922865 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:18.922871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:18.922982 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:18.923103 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:18.924647 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:18.925364 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:18.927087 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:18.927537 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:18.927677 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:18.931698 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:18.931704 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:18.931707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:18.931741 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:18.934432 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:18.935348 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:18.935356 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:18.935360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:18.935563 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:18.936307 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:18.936311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:18.936313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:18.936588 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:18.936708 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:18.937888 (+ 1180us) server_negotiation.cc:299] Negotiation successful
0620 02:52:18.938106 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":247,"threads_started":1}
I20260620 02:52:18.940120 3044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:18.920082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34209 (local address 127.0.97.1:55367)
0620 02:52:18.920779 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:18.920816 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:18.922582 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:18.923341 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:18.923350 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:18.923878 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:18.924476 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:18.924494 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:18.925530 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:18.925534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:18.926925 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:18.926944 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:18.927769 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:18.927777 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:18.928028 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:18.928783 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:18.928815 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:18.931430 (+ 2615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:18.934665 (+ 3235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:18.934673 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:18.934689 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:18.935165 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:18.935752 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:18.935756 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:18.935759 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:18.936110 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:18.936825 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:18.936836 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:18.937375 (+ 539us) client_negotiation.cc:769] Sending connection context
0620 02:52:18.937727 (+ 352us) client_negotiation.cc:240] Negotiation successful
0620 02:52:18.938345 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":268,"threads_started":1}
I20260620 02:52:18.942081 3041 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34209
I20260620 02:52:18.942453 3041 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:18.943388 3041 heartbeater.cc:511] Master 127.0.97.62:34209 requested a full tablet report, sending...
I20260620 02:52:18.946197 2852 ts_manager.cc:194] Registered new tserver with Master: abf40708e4a644118ae1151cf0a21b24 (127.0.97.1:39877)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:52:18.948637 2852 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55367
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:52:18.984220 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34209
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:19.311921 3047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:19.312270 3047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:19.312444 3047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:19.319926 3047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:19.320103 3047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:19.320277 3047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:52:19.330446 3047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:19.332288 3047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:19.334031 3047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:19.347855 3053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:19.347924 3052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:19.347947 3055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:19.348860 3047 server_base.cc:1034] running on GCE node
I20260620 02:52:19.349506 3047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:19.350529 3047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:19.351745 3047 hybrid_clock.cc:648] HybridClock initialized: now 1781923939351690 us; error 54 us; skew 500 ppm
I20260620 02:52:19.352123 3047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:19 dist-test-slave-8p8c krb5kdc[2807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:19.355944 3047 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:52:19.357800 3047 webserver.cc:460] Webserver started at http://127.0.97.2:43613/ using document root <none> and password file <none>
I20260620 02:52:19.358635 3047 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:19.358810 3047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:19.359145 3047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:19.362105 3047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/data/instance:
uuid: "a2d68e7328a940b5996fa477c3475a4c"
format_stamp: "Formatted at 2026-06-20 02:52:19 on dist-test-slave-8p8c"
server_key: "2251c78129a341ec18d5559c3525cce7"
server_key_iv: "eeb9ec9d8d304bea0a7d97352b008488"
server_key_version: "encryptionkey@0"
I20260620 02:52:19.362967 3047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a2d68e7328a940b5996fa477c3475a4c"
format_stamp: "Formatted at 2026-06-20 02:52:19 on dist-test-slave-8p8c"
server_key: "2251c78129a341ec18d5559c3525cce7"
server_key_iv: "eeb9ec9d8d304bea0a7d97352b008488"
server_key_version: "encryptionkey@0"
I20260620 02:52:19.368125 3047 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:52:19.371767 3062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:19.373132 3047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:52:19.373308 3047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/wal
uuid: "a2d68e7328a940b5996fa477c3475a4c"
format_stamp: "Formatted at 2026-06-20 02:52:19 on dist-test-slave-8p8c"
server_key: "2251c78129a341ec18d5559c3525cce7"
server_key_iv: "eeb9ec9d8d304bea0a7d97352b008488"
server_key_version: "encryptionkey@0"
I20260620 02:52:19.373515 3047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:19.404711 3047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:19.409242 3047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:19.409564 3047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:19.410629 3047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:52:19.412011 3047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:19.412112 3047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:19.412266 3047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:19.412345 3047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:19.447204 3047 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35995
I20260620 02:52:19.447247 3174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35995 every 8 connection(s)
I20260620 02:52:19.448498 3047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:52:19.453750 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 3047
I20260620 02:52:19.453964 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:52:19.454571 388 external_mini_cluster.cc:1428] Setting key 087bedab03896bc632ff7fb61f0fe6cd
Jun 20 02:52:19 dist-test-slave-8p8c krb5kdc[2807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:19.474426 3177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:19.451231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36661 (local address 127.0.97.62:34209)
0620 02:52:19.451839 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:19.451843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:19.452933 (+ 1090us) server_negotiation.cc:407] Connection header received
0620 02:52:19.455754 (+ 2821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:19.455759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:19.455840 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:19.456018 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:19.458335 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:19.459194 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:19.461742 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:19.462247 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:19.462351 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:19.466848 (+ 4497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:19.466857 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:19.466864 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:19.466909 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:19.470867 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:19.471716 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:19.471724 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:19.471730 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:19.471904 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:19.472399 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:19.472405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:19.472410 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:19.472770 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:19.472930 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:19.473646 (+ 716us) server_negotiation.cc:299] Negotiation successful
0620 02:52:19.473912 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":369,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:52:19.475381 3178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:19.451931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34209 (local address 127.0.97.2:36661)
0620 02:52:19.452690 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:19.452735 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:19.455400 (+ 2665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:19.456428 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:19.456445 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:19.457249 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:19.458115 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:19.458145 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:19.459357 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:19.459365 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:19.461583 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:19.461601 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:19.462519 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:19.462527 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:19.462853 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:19.463669 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:19.463703 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:19.466603 (+ 2900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:19.471025 (+ 4422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:19.471033 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:19.471084 (+ 51us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:19.471576 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:19.472049 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:19.472053 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:19.472056 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:19.472260 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:19.472922 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:19.472929 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:19.473269 (+ 340us) client_negotiation.cc:769] Sending connection context
0620 02:52:19.473515 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 02:52:19.474053 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":300,"threads_started":1}
I20260620 02:52:19.477154 3175 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 02:52:19.477674 3175 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:19.478559 3175 heartbeater.cc:511] Master 127.0.97.62:34209 requested a full tablet report, sending...
I20260620 02:52:19.480137 2852 ts_manager.cc:194] Registered new tserver with Master: a2d68e7328a940b5996fa477c3475a4c (127.0.97.2:35995)
I20260620 02:52:19.481390 2852 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36661
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:52:19.502744 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34209
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:19.845863 3181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:19.846175 3181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:19.846341 3181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:19.855144 3181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:19.855296 3181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:19.855496 3181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:52:19.866045 3181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:19.867550 3181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:19.868942 3181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:19.883482 3186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:19.883523 3187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:19.883661 3189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:19.884614 3181 server_base.cc:1034] running on GCE node
I20260620 02:52:19.885166 3181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:19.886087 3181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:19.887324 3181 hybrid_clock.cc:648] HybridClock initialized: now 1781923939887238 us; error 68 us; skew 500 ppm
I20260620 02:52:19.887733 3181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:19 dist-test-slave-8p8c krb5kdc[2807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:19.891296 3181 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:52:19.892921 3181 webserver.cc:460] Webserver started at http://127.0.97.3:37901/ using document root <none> and password file <none>
I20260620 02:52:19.893623 3181 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:19.893771 3181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:19.894120 3181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:19.897114 3181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/data/instance:
uuid: "4b45fc03645a41f8aff2c5a11a8f2745"
format_stamp: "Formatted at 2026-06-20 02:52:19 on dist-test-slave-8p8c"
server_key: "fd139ffae186671740ce8b7314230629"
server_key_iv: "168f5d480fef8c2a3b4722efc2982cea"
server_key_version: "encryptionkey@0"
I20260620 02:52:19.897908 3181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4b45fc03645a41f8aff2c5a11a8f2745"
format_stamp: "Formatted at 2026-06-20 02:52:19 on dist-test-slave-8p8c"
server_key: "fd139ffae186671740ce8b7314230629"
server_key_iv: "168f5d480fef8c2a3b4722efc2982cea"
server_key_version: "encryptionkey@0"
I20260620 02:52:19.903200 3181 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 02:52:19.906757 3196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:19.908169 3181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 02:52:19.908351 3181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/wal
uuid: "4b45fc03645a41f8aff2c5a11a8f2745"
format_stamp: "Formatted at 2026-06-20 02:52:19 on dist-test-slave-8p8c"
server_key: "fd139ffae186671740ce8b7314230629"
server_key_iv: "168f5d480fef8c2a3b4722efc2982cea"
server_key_version: "encryptionkey@0"
I20260620 02:52:19.908567 3181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:19.929173 3181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:19.933568 3181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:19.933863 3181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:19.934903 3181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:52:19.936260 3181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:19.936379 3181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:19.936527 3181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:19.936597 3181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:19.952721 3041 heartbeater.cc:503] Master 127.0.97.62:34209 was elected leader, sending a full tablet report...
I20260620 02:52:19.972478 3181 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35743
I20260620 02:52:19.972649 3308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35743 every 8 connection(s)
I20260620 02:52:19.973902 3181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:52:19.982929 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 3181
I20260620 02:52:19.983165 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:52:19.983621 388 external_mini_cluster.cc:1428] Setting key d739b5d0cbac4d3d6ae4a1593e092c03
Jun 20 02:52:19 dist-test-slave-8p8c krb5kdc[2807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:19.996775 3311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:19.977051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46729 (local address 127.0.97.62:34209)
0620 02:52:19.977571 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:19.977577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:19.978743 (+ 1166us) server_negotiation.cc:407] Connection header received
0620 02:52:19.980583 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:19.980589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:19.980689 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:19.980820 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:19.982833 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:19.984028 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:19.986149 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:19.986814 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:19.986980 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:19.990737 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:19.990742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:19.990745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:19.990778 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:19.993375 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:19.994167 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:19.994172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:19.994174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:19.994304 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:19.994981 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:19.995021 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:19.995024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:19.995288 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:19.995409 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:19.996330 (+ 921us) server_negotiation.cc:299] Negotiation successful
0620 02:52:19.996503 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":236,"threads_started":1}
I20260620 02:52:19.997946 3312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:19.977742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34209 (local address 127.0.97.3:46729)
0620 02:52:19.978592 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:19.978631 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:19.980301 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:19.981079 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:19.981087 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:19.981659 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:19.982610 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:19.982662 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:19.984192 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:19.984204 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:19.985997 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:19.986010 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:19.987075 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:19.987083 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:19.987375 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:19.988024 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:19.988052 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:19.990495 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:19.993512 (+ 3017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:19.993522 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:19.993538 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:19.994001 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:19.994510 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:19.994514 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:19.994518 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:19.994704 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:19.995441 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:19.995448 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:19.995784 (+ 336us) client_negotiation.cc:769] Sending connection context
0620 02:52:19.996065 (+ 281us) client_negotiation.cc:240] Negotiation successful
0620 02:52:19.996516 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":292,"threads_started":1}
I20260620 02:52:19.999604 3309 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34209
I20260620 02:52:19.999917 3309 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:20.000643 3309 heartbeater.cc:511] Master 127.0.97.62:34209 requested a full tablet report, sending...
I20260620 02:52:20.002117 2852 ts_manager.cc:194] Registered new tserver with Master: 4b45fc03645a41f8aff2c5a11a8f2745 (127.0.97.3:35743)
I20260620 02:52:20.002972 2852 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46729
I20260620 02:52:20.012285 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 02:52:20.042658 3311 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:52:20.040128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36472 (local address 127.0.97.62:34209)
0620 02:52:20.040314 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:20.040318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:20.040778 (+ 460us) server_negotiation.cc:407] Connection header received
0620 02:52:20.040886 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:20.040890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:20.040965 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:20.041062 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:20.042363 (+ 1301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36472: BlockingRecv error: recv got EOF from 127.0.0.1:36472 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 02:52:20.045557 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 2913
W20260620 02:52:20.048775 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 2913: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:39877: server requires authentication, but client does not have Kerberos credentials available
I20260620 02:52:20.049010 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 2913
I20260620 02:52:20.064764 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 3047
W20260620 02:52:20.067730 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 3047: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:35995: server requires authentication, but client does not have Kerberos credentials available
I20260620 02:52:20.067905 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 3047
I20260620 02:52:20.084975 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 3181
W20260620 02:52:20.087946 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 3181: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:35743: server requires authentication, but client does not have Kerberos credentials available
I20260620 02:52:20.088101 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 3181
I20260620 02:52:20.102946 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 2820
I20260620 02:52:20.104203 2882 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:20.207553 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 2820
2026-06-20T02:52:20Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3613 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:52:20 dist-test-slave-8p8c krb5kdc[3327](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:52:20 dist-test-slave-8p8c krb5kdc[3327](info): set up 2 sockets
Jun 20 02:52:20 dist-test-slave-8p8c krb5kdc[3327](info): commencing operation
krb5kdc: starting...
W20260620 02:52:21.318935 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:52:21 dist-test-slave-8p8c krb5kdc[3327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:52:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:52:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:52:21.530929 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36097
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:21.855691 3340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:21.856001 3340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:21.856089 3340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:21.863472 3340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:52:21.863600 3340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:21.863646 3340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:21.863695 3340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:52:21.863741 3340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:52:21.873638 3340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:21.874971 3340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:21.876269 3340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:21.889768 3345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:21.889629 3348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:21.889719 3346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:21.890138 3340 server_base.cc:1034] running on GCE node
I20260620 02:52:21.891302 3340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:21.892827 3340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:21.894081 3340 hybrid_clock.cc:648] HybridClock initialized: now 1781923941894025 us; error 49 us; skew 500 ppm
I20260620 02:52:21.894527 3340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:21 dist-test-slave-8p8c krb5kdc[3327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:21.903514 3340 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:52:21.905179 3340 webserver.cc:460] Webserver started at http://127.0.97.62:39313/ using document root <none> and password file <none>
I20260620 02:52:21.905870 3340 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:21.905973 3340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:21.906217 3340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:21.909319 3340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "a2d911e29611487a91a8e7c22d153eac"
format_stamp: "Formatted at 2026-06-20 02:52:21 on dist-test-slave-8p8c"
server_key: "d738b04ba675384b69be75fb35bb5879"
server_key_iv: "d40ef7b618d89fcd0fc522749d26959d"
server_key_version: "encryptionkey@0"
I20260620 02:52:21.910058 3340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "a2d911e29611487a91a8e7c22d153eac"
format_stamp: "Formatted at 2026-06-20 02:52:21 on dist-test-slave-8p8c"
server_key: "d738b04ba675384b69be75fb35bb5879"
server_key_iv: "d40ef7b618d89fcd0fc522749d26959d"
server_key_version: "encryptionkey@0"
I20260620 02:52:21.915472 3340 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 02:52:21.919420 3355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:21.921209 3340 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260620 02:52:21.921365 3340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "a2d911e29611487a91a8e7c22d153eac"
format_stamp: "Formatted at 2026-06-20 02:52:21 on dist-test-slave-8p8c"
server_key: "d738b04ba675384b69be75fb35bb5879"
server_key_iv: "d40ef7b618d89fcd0fc522749d26959d"
server_key_version: "encryptionkey@0"
I20260620 02:52:21.921546 3340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:21.946677 3340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:21.951784 3340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:21.952054 3340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:21.972087 3340 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36097
I20260620 02:52:21.972177 3406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36097 every 8 connection(s)
I20260620 02:52:21.973521 3340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:52:21.977972 3407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:21.979954 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 3340
I20260620 02:52:21.980201 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:52:21.981081 388 external_mini_cluster.cc:1428] Setting key fd129a618c5f126143945fd11f917253
I20260620 02:52:21.991981 3407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2d911e29611487a91a8e7c22d153eac: Bootstrap starting.
Jun 20 02:52:21 dist-test-slave-8p8c krb5kdc[3327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:21.995150 3407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2d911e29611487a91a8e7c22d153eac: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:21.996186 3407 log.cc:824] T 00000000000000000000000000000000 P a2d911e29611487a91a8e7c22d153eac: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:21.999230 3407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2d911e29611487a91a8e7c22d153eac: No bootstrap required, opened a new log
I20260620 02:52:22.003283 3410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:21.983256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59008 (local address 127.0.97.62:36097)
0620 02:52:21.984115 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:21.984130 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:21.984174 (+ 44us) server_negotiation.cc:407] Connection header received
0620 02:52:21.985718 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:21.985743 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:21.986185 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:21.986690 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:21.988150 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:21.989504 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:21.990840 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:21.991527 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:21.991727 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:21.995703 (+ 3976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:21.995712 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:21.995728 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:21.995769 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:21.999022 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:21.999792 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:21.999799 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:21.999805 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:21.999965 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:22.000356 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:22.000365 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:22.000367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:22.000903 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:22.001247 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:22.001647 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 02:52:22.002054 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":367,"threads_started":1}
I20260620 02:52:22.007223 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:52:22.007488 3407 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2d911e29611487a91a8e7c22d153eac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d911e29611487a91a8e7c22d153eac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36097 } }
I20260620 02:52:22.007856 3407 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2d911e29611487a91a8e7c22d153eac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:22.007956 3407 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2d911e29611487a91a8e7c22d153eac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2d911e29611487a91a8e7c22d153eac, State: Initialized, Role: FOLLOWER
I20260620 02:52:22.008502 3407 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2d911e29611487a91a8e7c22d153eac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d911e29611487a91a8e7c22d153eac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36097 } }
I20260620 02:52:22.008735 3407 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2d911e29611487a91a8e7c22d153eac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:52:22.008869 3407 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2d911e29611487a91a8e7c22d153eac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:52:22.009030 3407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2d911e29611487a91a8e7c22d153eac [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:22.011197 3407 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2d911e29611487a91a8e7c22d153eac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d911e29611487a91a8e7c22d153eac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36097 } }
I20260620 02:52:22.011684 3407 leader_election.cc:302] T 00000000000000000000000000000000 P a2d911e29611487a91a8e7c22d153eac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2d911e29611487a91a8e7c22d153eac; no voters:
I20260620 02:52:22.012341 3407 leader_election.cc:288] T 00000000000000000000000000000000 P a2d911e29611487a91a8e7c22d153eac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:52:22.012437 3412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2d911e29611487a91a8e7c22d153eac [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:22.015364 3407 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2d911e29611487a91a8e7c22d153eac [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:52:22.019527 3412 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2d911e29611487a91a8e7c22d153eac [term 1 LEADER]: Becoming Leader. State: Replica: a2d911e29611487a91a8e7c22d153eac, State: Running, Role: LEADER
I20260620 02:52:22.020426 3412 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2d911e29611487a91a8e7c22d153eac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d911e29611487a91a8e7c22d153eac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36097 } }
I20260620 02:52:22.023928 3414 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2d911e29611487a91a8e7c22d153eac [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2d911e29611487a91a8e7c22d153eac. Latest consensus state: current_term: 1 leader_uuid: "a2d911e29611487a91a8e7c22d153eac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d911e29611487a91a8e7c22d153eac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36097 } } }
I20260620 02:52:22.024292 3414 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2d911e29611487a91a8e7c22d153eac [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:22.024847 3413 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2d911e29611487a91a8e7c22d153eac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2d911e29611487a91a8e7c22d153eac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d911e29611487a91a8e7c22d153eac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36097 } } }
I20260620 02:52:22.025185 3413 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2d911e29611487a91a8e7c22d153eac [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:22.029637 3428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:52:22.042074 3428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:52:22.051656 3410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:22.042630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59016 (local address 127.0.97.62:36097)
0620 02:52:22.042836 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:22.042844 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:22.043505 (+ 661us) server_negotiation.cc:407] Connection header received
0620 02:52:22.043727 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:22.043733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:22.043811 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:22.043928 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:22.044909 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:22.045876 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:22.047663 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:22.048457 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:22.048616 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:22.049879 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:22.049884 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:22.049887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:52:22.050681 (+ 794us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:22.050776 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:52:22.050824 (+ 48us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:22.050937 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:22.051102 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 02:52:22.051202 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 02:52:22.053284 3428 catalog_manager.cc:1347] Generated new cluster ID: b495907701b44d3293afef91709fba82
I20260620 02:52:22.053421 3428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:52:22.094642 3428 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:52:22.096072 3428 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:52:22.105576 3428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2d911e29611487a91a8e7c22d153eac: Generated new TSK 0
I20260620 02:52:22.106760 3428 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:52:22.117307 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 3340
I20260620 02:52:22.118692 3402 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:22.217866 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 3340
2026-06-20T02:52:22Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2004 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:52:22 dist-test-slave-8p8c krb5kdc[3441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:52:22 dist-test-slave-8p8c krb5kdc[3441](info): set up 2 sockets
Jun 20 02:52:22 dist-test-slave-8p8c krb5kdc[3441](info): commencing operation
krb5kdc: starting...
W20260620 02:52:23.319417 388 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.002s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:52:23 dist-test-slave-8p8c krb5kdc[3441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:52:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:52:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:52:23.523383 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39219
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:23.846503 3454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:23.846903 3454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:23.847066 3454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:23.855489 3454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:52:23.855691 3454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:23.855814 3454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:23.855931 3454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:52:23.856040 3454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:52:23.867396 3454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:23.868963 3454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:23.870474 3454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:23.883488 3460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:23.883518 3462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:23.883556 3459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:23.884267 3454 server_base.cc:1034] running on GCE node
I20260620 02:52:23.885145 3454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:23.886739 3454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:23.888007 3454 hybrid_clock.cc:648] HybridClock initialized: now 1781923943887934 us; error 65 us; skew 500 ppm
I20260620 02:52:23.888393 3454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:23 dist-test-slave-8p8c krb5kdc[3441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:23.891945 3454 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:52:23.893533 3454 webserver.cc:460] Webserver started at http://127.0.97.62:40565/ using document root <none> and password file <none>
I20260620 02:52:23.894295 3454 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:23.894491 3454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:23.894845 3454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:23.898599 3454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "08a94beff22345d69d605b5c9c2b26bb"
format_stamp: "Formatted at 2026-06-20 02:52:23 on dist-test-slave-8p8c"
server_key: "cd1da5d9a6e7fab252ae0921a05e986a"
server_key_iv: "51cb4686b5decc882ede9e21421d068e"
server_key_version: "encryptionkey@0"
I20260620 02:52:23.899454 3454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "08a94beff22345d69d605b5c9c2b26bb"
format_stamp: "Formatted at 2026-06-20 02:52:23 on dist-test-slave-8p8c"
server_key: "cd1da5d9a6e7fab252ae0921a05e986a"
server_key_iv: "51cb4686b5decc882ede9e21421d068e"
server_key_version: "encryptionkey@0"
I20260620 02:52:23.905125 3454 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 02:52:23.908720 3469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:23.910236 3454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:52:23.910584 3454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "08a94beff22345d69d605b5c9c2b26bb"
format_stamp: "Formatted at 2026-06-20 02:52:23 on dist-test-slave-8p8c"
server_key: "cd1da5d9a6e7fab252ae0921a05e986a"
server_key_iv: "51cb4686b5decc882ede9e21421d068e"
server_key_version: "encryptionkey@0"
I20260620 02:52:23.910804 3454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:23.927734 3454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:23.932866 3454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:23.933187 3454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:23.953742 3454 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39219
I20260620 02:52:23.953771 3520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39219 every 8 connection(s)
I20260620 02:52:23.955633 3454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:52:23.960052 3521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:23.963248 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 3454
I20260620 02:52:23.963490 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:52:23.964344 388 external_mini_cluster.cc:1428] Setting key e7378ff38ccdd0987884230b8a74b240
I20260620 02:52:23.974104 3521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08a94beff22345d69d605b5c9c2b26bb: Bootstrap starting.
I20260620 02:52:23.977514 3521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08a94beff22345d69d605b5c9c2b26bb: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:23.978677 3521 log.cc:824] T 00000000000000000000000000000000 P 08a94beff22345d69d605b5c9c2b26bb: Log is configured to *not* fsync() on all Append() calls
Jun 20 02:52:23 dist-test-slave-8p8c krb5kdc[3441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:23.982175 3521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08a94beff22345d69d605b5c9c2b26bb: No bootstrap required, opened a new log
I20260620 02:52:23.992043 3524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:23.966842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38820 (local address 127.0.97.62:39219)
0620 02:52:23.967776 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:23.967790 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:23.967837 (+ 47us) server_negotiation.cc:407] Connection header received
0620 02:52:23.969779 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:23.969807 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:23.970267 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:23.970829 (+ 562us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:23.972413 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:23.973864 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:23.975630 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:23.976409 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:23.976646 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:23.982033 (+ 5387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:23.982042 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:23.982058 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:23.982099 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:23.987305 (+ 5206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:23.988215 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:23.988223 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:23.988231 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:23.988431 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:23.988881 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:23.988886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:23.988888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:23.989506 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:23.989879 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:23.990413 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 02:52:23.990903 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":413,"threads_started":1}
I20260620 02:52:23.992035 3521 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08a94beff22345d69d605b5c9c2b26bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08a94beff22345d69d605b5c9c2b26bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39219 } }
I20260620 02:52:23.992542 3521 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08a94beff22345d69d605b5c9c2b26bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:23.992673 3521 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08a94beff22345d69d605b5c9c2b26bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08a94beff22345d69d605b5c9c2b26bb, State: Initialized, Role: FOLLOWER
I20260620 02:52:23.993343 3521 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08a94beff22345d69d605b5c9c2b26bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08a94beff22345d69d605b5c9c2b26bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39219 } }
I20260620 02:52:23.993594 3521 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08a94beff22345d69d605b5c9c2b26bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:52:23.993765 3521 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08a94beff22345d69d605b5c9c2b26bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:52:23.993959 3521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08a94beff22345d69d605b5c9c2b26bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:23.996492 3521 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08a94beff22345d69d605b5c9c2b26bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08a94beff22345d69d605b5c9c2b26bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39219 } }
I20260620 02:52:23.997054 3521 leader_election.cc:302] T 00000000000000000000000000000000 P 08a94beff22345d69d605b5c9c2b26bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08a94beff22345d69d605b5c9c2b26bb; no voters:
I20260620 02:52:23.997687 3521 leader_election.cc:288] T 00000000000000000000000000000000 P 08a94beff22345d69d605b5c9c2b26bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:52:23.997851 3526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08a94beff22345d69d605b5c9c2b26bb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:23.998714 3526 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08a94beff22345d69d605b5c9c2b26bb [term 1 LEADER]: Becoming Leader. State: Replica: 08a94beff22345d69d605b5c9c2b26bb, State: Running, Role: LEADER
I20260620 02:52:23.999305 3526 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08a94beff22345d69d605b5c9c2b26bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08a94beff22345d69d605b5c9c2b26bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39219 } }
I20260620 02:52:23.999831 3521 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08a94beff22345d69d605b5c9c2b26bb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:52:24.002811 3527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08a94beff22345d69d605b5c9c2b26bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08a94beff22345d69d605b5c9c2b26bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08a94beff22345d69d605b5c9c2b26bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39219 } } }
I20260620 02:52:24.003154 3527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08a94beff22345d69d605b5c9c2b26bb [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:24.003618 3528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08a94beff22345d69d605b5c9c2b26bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08a94beff22345d69d605b5c9c2b26bb. Latest consensus state: current_term: 1 leader_uuid: "08a94beff22345d69d605b5c9c2b26bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08a94beff22345d69d605b5c9c2b26bb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39219 } } }
I20260620 02:52:24.003926 3528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08a94beff22345d69d605b5c9c2b26bb [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:24.004794 3533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:52:24.010605 3533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:52:24.021478 3533 catalog_manager.cc:1347] Generated new cluster ID: f83bff44a1644711b1cd4dd721fbe0b4
I20260620 02:52:24.021672 3533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:52:24.039240 3533 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:52:24.040833 3533 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:52:24.051017 3533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08a94beff22345d69d605b5c9c2b26bb: Generated new TSK 0
I20260620 02:52:24.052023 3533 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:52:24.099852 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39219
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:24.442026 3547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:24.442320 3547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:24.442499 3547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:24.450121 3547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:24.450238 3547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:24.450474 3547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:52:24.460510 3547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:24.461813 3547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:24.463215 3547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:24.476393 3553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:24.476397 3552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:24.476457 3555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:24.477420 3547 server_base.cc:1034] running on GCE node
I20260620 02:52:24.478132 3547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:24.479143 3547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:24.480319 3547 hybrid_clock.cc:648] HybridClock initialized: now 1781923944480251 us; error 57 us; skew 500 ppm
I20260620 02:52:24.480657 3547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:24 dist-test-slave-8p8c krb5kdc[3441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:24.483937 3547 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:52:24.485545 3547 webserver.cc:460] Webserver started at http://127.0.97.1:38157/ using document root <none> and password file <none>
I20260620 02:52:24.486267 3547 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:24.486466 3547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:24.486848 3547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:24.489885 3547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/data/instance:
uuid: "dd461396e48e4d508db508839e9e2c8b"
format_stamp: "Formatted at 2026-06-20 02:52:24 on dist-test-slave-8p8c"
server_key: "6f972809d110db77b0bf13f5831fb428"
server_key_iv: "3c8a58eaf6d52c61f54be4e1feaef942"
server_key_version: "encryptionkey@0"
I20260620 02:52:24.490764 3547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/wal/instance:
uuid: "dd461396e48e4d508db508839e9e2c8b"
format_stamp: "Formatted at 2026-06-20 02:52:24 on dist-test-slave-8p8c"
server_key: "6f972809d110db77b0bf13f5831fb428"
server_key_iv: "3c8a58eaf6d52c61f54be4e1feaef942"
server_key_version: "encryptionkey@0"
I20260620 02:52:24.496394 3547 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 02:52:24.499737 3562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:24.501238 3547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:52:24.501402 3547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/wal
uuid: "dd461396e48e4d508db508839e9e2c8b"
format_stamp: "Formatted at 2026-06-20 02:52:24 on dist-test-slave-8p8c"
server_key: "6f972809d110db77b0bf13f5831fb428"
server_key_iv: "3c8a58eaf6d52c61f54be4e1feaef942"
server_key_version: "encryptionkey@0"
I20260620 02:52:24.501579 3547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:24.514341 3547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:24.518918 3547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:24.519223 3547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:24.520166 3547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:52:24.521441 3547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:24.521515 3547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:24.521634 3547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:24.521692 3547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:24.555162 3547 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46537
I20260620 02:52:24.555284 3674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46537 every 8 connection(s)
I20260620 02:52:24.556419 3547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:52:24.560017 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 3547
I20260620 02:52:24.560249 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:52:24.561142 388 external_mini_cluster.cc:1428] Setting key 45bd0223fb3af15d9a9539dfa9359e02
Jun 20 02:52:24 dist-test-slave-8p8c krb5kdc[3441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:24.581386 3677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:24.559013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58861 (local address 127.0.97.62:39219)
0620 02:52:24.559460 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:24.559500 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:24.560590 (+ 1090us) server_negotiation.cc:407] Connection header received
0620 02:52:24.563139 (+ 2549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:24.563144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:24.563230 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:24.563342 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:24.565709 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:24.566477 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:24.569286 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:24.570070 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:24.570241 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:24.573668 (+ 3427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:24.573678 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:24.573686 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:24.573730 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:24.577578 (+ 3848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:24.578783 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:24.578787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:24.578789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:24.578917 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:24.579483 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:24.579487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:24.579490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:24.579761 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:24.579878 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:24.580923 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0620 02:52:24.581138 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":252,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:52:24.583035 3678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:24.559505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39219 (local address 127.0.97.1:58861)
0620 02:52:24.560390 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:24.560447 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:24.562811 (+ 2364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:24.563686 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:24.563700 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:24.564555 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:24.565503 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:24.565530 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:24.566650 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:24.566659 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:24.569096 (+ 2437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:24.569122 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:24.570319 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:24.570324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:24.570607 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:24.571308 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:24.571331 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:24.573446 (+ 2115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:24.577746 (+ 4300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:24.577758 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:24.577782 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:24.578643 (+ 861us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:24.579040 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:24.579050 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:24.579056 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:24.579321 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:24.579915 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:24.579927 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:24.580437 (+ 510us) client_negotiation.cc:769] Sending connection context
0620 02:52:24.580812 (+ 375us) client_negotiation.cc:240] Negotiation successful
0620 02:52:24.581478 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":306,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 02:52:24.584719 3675 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39219
I20260620 02:52:24.585065 3675 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:24.585794 3675 heartbeater.cc:511] Master 127.0.97.62:39219 requested a full tablet report, sending...
I20260620 02:52:24.588032 3486 ts_manager.cc:194] Registered new tserver with Master: dd461396e48e4d508db508839e9e2c8b (127.0.97.1:46537)
I20260620 02:52:24.589843 3486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58861
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:52:24.609457 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39219
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:24.940315 3681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:24.940605 3681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:24.940699 3681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:24.948311 3681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:24.948439 3681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:24.948556 3681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:52:24.962433 3681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:24.964135 3681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:24.965799 3681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:24.982494 3686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:24.982564 3689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:24.982538 3687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:24.988631 3681 server_base.cc:1034] running on GCE node
I20260620 02:52:24.989291 3681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:24.990301 3681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:24.991636 3681 hybrid_clock.cc:648] HybridClock initialized: now 1781923944991561 us; error 70 us; skew 500 ppm
I20260620 02:52:24.992012 3681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:24 dist-test-slave-8p8c krb5kdc[3441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:24.995517 3681 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:52:24.997162 3681 webserver.cc:460] Webserver started at http://127.0.97.2:42153/ using document root <none> and password file <none>
I20260620 02:52:24.997968 3681 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:24.998071 3681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:24.998335 3681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:25.001333 3681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/data/instance:
uuid: "18aca3b67fa945ce9bfd2f4d14e35dba"
format_stamp: "Formatted at 2026-06-20 02:52:24 on dist-test-slave-8p8c"
server_key: "88467b1feb7c87228282b5f293af873f"
server_key_iv: "6be9a1424caa88ec931f7e10515ba01e"
server_key_version: "encryptionkey@0"
I20260620 02:52:25.002089 3681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/wal/instance:
uuid: "18aca3b67fa945ce9bfd2f4d14e35dba"
format_stamp: "Formatted at 2026-06-20 02:52:24 on dist-test-slave-8p8c"
server_key: "88467b1feb7c87228282b5f293af873f"
server_key_iv: "6be9a1424caa88ec931f7e10515ba01e"
server_key_version: "encryptionkey@0"
I20260620 02:52:25.007877 3681 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 02:52:25.011806 3696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:25.013815 3681 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260620 02:52:25.014032 3681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/wal
uuid: "18aca3b67fa945ce9bfd2f4d14e35dba"
format_stamp: "Formatted at 2026-06-20 02:52:24 on dist-test-slave-8p8c"
server_key: "88467b1feb7c87228282b5f293af873f"
server_key_iv: "6be9a1424caa88ec931f7e10515ba01e"
server_key_version: "encryptionkey@0"
I20260620 02:52:25.014290 3681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:25.039844 3681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:25.044457 3681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:25.044782 3681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:25.045845 3681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:52:25.047406 3681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:25.047482 3681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:25.047614 3681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:25.047672 3681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:25.083634 3681 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34201
I20260620 02:52:25.083655 3808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34201 every 8 connection(s)
I20260620 02:52:25.084967 3681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:52:25.089745 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 3681
I20260620 02:52:25.089926 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:52:25.090351 388 external_mini_cluster.cc:1428] Setting key a26c5135c156ad08a8a89fd8b985ad15
Jun 20 02:52:25 dist-test-slave-8p8c krb5kdc[3441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:25.108474 3811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:25.087614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45405 (local address 127.0.97.62:39219)
0620 02:52:25.088306 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:25.088316 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:25.089310 (+ 994us) server_negotiation.cc:407] Connection header received
0620 02:52:25.091396 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:25.091400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:25.091464 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:25.091561 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:25.093231 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:25.094701 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:25.096610 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:25.097472 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:25.097630 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:25.101890 (+ 4260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:25.101898 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:25.101904 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:25.101951 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:25.105048 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:25.106022 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:25.106028 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:25.106031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:25.106188 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:25.106688 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:25.106693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:25.106699 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:25.107021 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:25.107158 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:25.107857 (+ 699us) server_negotiation.cc:299] Negotiation successful
0620 02:52:25.108091 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":320,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:52:25.109578 3812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:25.088381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39219 (local address 127.0.97.2:45405)
0620 02:52:25.089144 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:25.089191 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:25.091116 (+ 1925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:25.091865 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:25.091877 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:25.092472 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:25.093048 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:25.093067 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:25.095048 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:25.095052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:25.096452 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:25.096464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:25.097709 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:25.097714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:25.097930 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:25.098727 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:25.098751 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:25.101620 (+ 2869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:25.105207 (+ 3587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:25.105216 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:25.105235 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:25.105871 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:25.106300 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:25.106305 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:25.106309 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:25.106560 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:25.107141 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:25.107149 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:25.107503 (+ 354us) client_negotiation.cc:769] Sending connection context
0620 02:52:25.107771 (+ 268us) client_negotiation.cc:240] Negotiation successful
0620 02:52:25.108326 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":267,"threads_started":1}
I20260620 02:52:25.111287 3809 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39219
I20260620 02:52:25.111639 3809 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:25.112548 3809 heartbeater.cc:511] Master 127.0.97.62:39219 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 02:52:25.114229 3486 ts_manager.cc:194] Registered new tserver with Master: 18aca3b67fa945ce9bfd2f4d14e35dba (127.0.97.2:34201)
I20260620 02:52:25.115191 3486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:52:25.138716 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39219
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:25.488601 3815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:25.488974 3815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:25.489145 3815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:25.496899 3815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:25.497087 3815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:25.497273 3815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:52:25.508006 3815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:25.509394 3815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:25.510775 3815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:25.525141 3820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:25.525125 3823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:25.525102 3821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:25.526543 3815 server_base.cc:1034] running on GCE node
I20260620 02:52:25.527274 3815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:25.528352 3815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:25.529654 3815 hybrid_clock.cc:648] HybridClock initialized: now 1781923945529615 us; error 28 us; skew 500 ppm
I20260620 02:52:25.530021 3815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:25 dist-test-slave-8p8c krb5kdc[3441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:25.533541 3815 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:52:25.535392 3815 webserver.cc:460] Webserver started at http://127.0.97.3:43891/ using document root <none> and password file <none>
I20260620 02:52:25.536154 3815 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:25.536304 3815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:25.536621 3815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:25.539700 3815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/data/instance:
uuid: "50a1a3a4074e49e68836b9913da94eec"
format_stamp: "Formatted at 2026-06-20 02:52:25 on dist-test-slave-8p8c"
server_key: "8c0a745e937da7a0b80b1b3c7cbfaf16"
server_key_iv: "b818bee3e6ad9013bc83ba4deed7556b"
server_key_version: "encryptionkey@0"
I20260620 02:52:25.540764 3815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/wal/instance:
uuid: "50a1a3a4074e49e68836b9913da94eec"
format_stamp: "Formatted at 2026-06-20 02:52:25 on dist-test-slave-8p8c"
server_key: "8c0a745e937da7a0b80b1b3c7cbfaf16"
server_key_iv: "b818bee3e6ad9013bc83ba4deed7556b"
server_key_version: "encryptionkey@0"
I20260620 02:52:25.546347 3815 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 02:52:25.550696 3830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:25.552379 3815 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260620 02:52:25.552603 3815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/wal
uuid: "50a1a3a4074e49e68836b9913da94eec"
format_stamp: "Formatted at 2026-06-20 02:52:25 on dist-test-slave-8p8c"
server_key: "8c0a745e937da7a0b80b1b3c7cbfaf16"
server_key_iv: "b818bee3e6ad9013bc83ba4deed7556b"
server_key_version: "encryptionkey@0"
I20260620 02:52:25.552835 3815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:25.572698 3815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:25.577713 3815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:25.578068 3815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:25.579242 3815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:52:25.580891 3815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:25.581025 3815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:25.581195 3815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:25.581277 3815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:25.592887 3675 heartbeater.cc:503] Master 127.0.97.62:39219 was elected leader, sending a full tablet report...
I20260620 02:52:25.615952 3815 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38155
I20260620 02:52:25.615971 3942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38155 every 8 connection(s)
I20260620 02:52:25.617568 3815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:52:25.619668 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 3815
I20260620 02:52:25.619860 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781923914418765-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:52:25.620350 388 external_mini_cluster.cc:1428] Setting key a6205e74b9578d8a922131165695853c
Jun 20 02:52:25 dist-test-slave-8p8c krb5kdc[3441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:25.641422 3945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:25.621091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40941 (local address 127.0.97.62:39219)
0620 02:52:25.621648 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:25.621653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:25.622999 (+ 1346us) server_negotiation.cc:407] Connection header received
0620 02:52:25.625351 (+ 2352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:25.625356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:25.625450 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:25.625564 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:25.627195 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:25.627973 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:25.629508 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:25.629934 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:25.630060 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:25.634246 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:25.634252 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:25.634255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:25.634290 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:25.637496 (+ 3206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:25.638602 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:25.638606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:25.638608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:25.638731 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:25.639372 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:25.639378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:25.639381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:25.639716 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:25.639840 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:25.640702 (+ 862us) server_negotiation.cc:299] Negotiation successful
0620 02:52:25.641150 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":650,"threads_started":1}
I20260620 02:52:25.642256 3946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:25.621807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39219 (local address 127.0.97.3:40941)
0620 02:52:25.622867 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:25.622916 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:25.625049 (+ 2133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:25.625814 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:25.625823 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:25.626436 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:25.627033 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:25.627051 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:25.628131 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:25.628135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:25.629368 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:25.629380 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:25.630175 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:25.630181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:25.630571 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:25.631288 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:25.631311 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:25.633984 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:25.637637 (+ 3653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:25.637644 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:25.637659 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:25.638452 (+ 793us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:25.638898 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:25.638903 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:25.638906 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:25.639145 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:25.639871 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:25.639878 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:25.640276 (+ 398us) client_negotiation.cc:769] Sending connection context
0620 02:52:25.640543 (+ 267us) client_negotiation.cc:240] Negotiation successful
0620 02:52:25.641004 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"thread_start_us":416,"threads_started":1}
I20260620 02:52:25.643915 3943 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39219
I20260620 02:52:25.644271 3943 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:25.645287 3943 heartbeater.cc:511] Master 127.0.97.62:39219 requested a full tablet report, sending...
I20260620 02:52:25.646931 3486 ts_manager.cc:194] Registered new tserver with Master: 50a1a3a4074e49e68836b9913da94eec (127.0.97.3:38155)
I20260620 02:52:25.647926 3486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40941
I20260620 02:52:25.649552 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 02:52:25 dist-test-slave-8p8c krb5kdc[3441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923945, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 02:52:25 dist-test-slave-8p8c krb5kdc[3441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923945, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:25.703833 3945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:25.686329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38826 (local address 127.0.97.62:39219)
0620 02:52:25.686614 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:25.686619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:25.687054 (+ 435us) server_negotiation.cc:407] Connection header received
0620 02:52:25.687373 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:25.687378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:25.687478 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:25.687580 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:25.689261 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:25.690210 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:25.691822 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:25.692307 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:25.692421 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:25.697587 (+ 5166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:25.697594 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:25.697598 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:25.697641 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:25.700982 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:25.701715 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:25.701720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:25.701722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:25.701854 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:25.702548 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:25.702552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:25.702554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:25.702890 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:25.703086 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:25.703418 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 02:52:25.703602 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
W20260620 02:52:25.704433 3486 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38826
I20260620 02:52:25.706663 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 3547
Jun 20 02:52:25 dist-test-slave-8p8c krb5kdc[3441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923945, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:52:25.729130 3955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:25.708090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46636 (local address 127.0.97.1:46537)
0620 02:52:25.708854 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:25.708864 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:25.708895 (+ 31us) server_negotiation.cc:407] Connection header received
0620 02:52:25.708962 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:25.708970 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:25.709285 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:25.709416 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:25.710825 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:25.712257 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:25.713626 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:25.714264 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:25.714441 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:25.720810 (+ 6369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:25.720832 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:25.720858 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:25.720929 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:25.725812 (+ 4883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:25.726764 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:25.726775 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:25.726779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:25.726937 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:25.727368 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:25.727375 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:25.727379 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:25.727776 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:25.728101 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:25.728449 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 02:52:25.728741 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":361,"threads_started":1}
W20260620 02:52:25.730613 3670 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46636
W20260620 02:52:25.731880 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 3547: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:52:25.732079 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 3547
I20260620 02:52:25.748227 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 3681
Jun 20 02:52:25 dist-test-slave-8p8c krb5kdc[3441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923945, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:52:25.766336 3956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:25.749237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37490 (local address 127.0.97.2:34201)
0620 02:52:25.749811 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:25.749819 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:25.749849 (+ 30us) server_negotiation.cc:407] Connection header received
0620 02:52:25.749913 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:25.749920 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:25.750181 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:25.750328 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:25.751638 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:25.752977 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:25.754455 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:25.755251 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:25.755385 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:25.760535 (+ 5150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:25.760548 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:25.760565 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:25.760609 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:25.763615 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:25.764319 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:25.764328 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:25.764330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:25.764494 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:25.764885 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:25.764890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:25.764892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:25.765284 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:25.765527 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:25.765848 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 02:52:25.766053 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":300,"threads_started":1}
W20260620 02:52:25.767732 3804 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37490
W20260620 02:52:25.768862 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 3681: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:52:25.769061 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 3681
I20260620 02:52:25.784690 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 3815
Jun 20 02:52:25 dist-test-slave-8p8c krb5kdc[3441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923945, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:52:25.802039 3957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:25.785627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51246 (local address 127.0.97.3:38155)
0620 02:52:25.786206 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:25.786213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:25.786243 (+ 30us) server_negotiation.cc:407] Connection header received
0620 02:52:25.786302 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:25.786309 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:25.786614 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:25.786791 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:25.788124 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:25.789175 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:25.790515 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:25.791086 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:25.791230 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:25.796256 (+ 5026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:25.796269 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:25.796285 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:25.796323 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:25.799364 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:25.800220 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:25.800228 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:25.800230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:25.800353 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:25.800668 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:25.800672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:25.800674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:25.801010 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:25.801376 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:25.801557 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 02:52:25.801752 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":275,"threads_started":1}
W20260620 02:52:25.803468 3938 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51246
W20260620 02:52:25.804603 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 3815: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:52:25.804817 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 3815
I20260620 02:52:25.821668 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 3454
I20260620 02:52:25.823060 3516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:25.925918 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 3454
2026-06-20T02:52:25Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3716 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:52:26 dist-test-slave-8p8c krb5kdc[3961](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:52:26 dist-test-slave-8p8c krb5kdc[3961](info): set up 2 sockets
Jun 20 02:52:26 dist-test-slave-8p8c krb5kdc[3961](info): commencing operation
krb5kdc: starting...
W20260620 02:52:27.047379 388 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:52:27 dist-test-slave-8p8c krb5kdc[3961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:52:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:52:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:52:27.243903 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40693
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40693
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:27.559604 3974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:27.560032 3974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:27.560140 3974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:27.567437 3974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:52:27.567543 3974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:27.567588 3974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:27.567651 3974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:52:27.567693 3974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:52:27.577210 3974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:27.578452 3974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:27.579654 3974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:27.591140 3979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:27.591059 3982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:27.591112 3980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:27.592131 3974 server_base.cc:1034] running on GCE node
I20260620 02:52:27.592934 3974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:27.594303 3974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:27.595634 3974 hybrid_clock.cc:648] HybridClock initialized: now 1781923947595579 us; error 44 us; skew 500 ppm
I20260620 02:52:27.596012 3974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:27 dist-test-slave-8p8c krb5kdc[3961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:27.605497 3974 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:52:27.607165 3974 webserver.cc:460] Webserver started at http://127.0.97.62:39933/ using document root <none> and password file <none>
I20260620 02:52:27.607893 3974 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:27.608048 3974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:27.608371 3974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:27.611476 3974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "0c61e05335fb4e6289bf0f90003f7228"
format_stamp: "Formatted at 2026-06-20 02:52:27 on dist-test-slave-8p8c"
server_key: "db26fe50385242fa81a0721b5f657c01"
server_key_iv: "03a6cc98a131e465a2037ae114ed4839"
server_key_version: "encryptionkey@0"
I20260620 02:52:27.612259 3974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "0c61e05335fb4e6289bf0f90003f7228"
format_stamp: "Formatted at 2026-06-20 02:52:27 on dist-test-slave-8p8c"
server_key: "db26fe50385242fa81a0721b5f657c01"
server_key_iv: "03a6cc98a131e465a2037ae114ed4839"
server_key_version: "encryptionkey@0"
I20260620 02:52:27.617352 3974 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:52:27.621059 3989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:27.622687 3974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 02:52:27.622893 3974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "0c61e05335fb4e6289bf0f90003f7228"
format_stamp: "Formatted at 2026-06-20 02:52:27 on dist-test-slave-8p8c"
server_key: "db26fe50385242fa81a0721b5f657c01"
server_key_iv: "03a6cc98a131e465a2037ae114ed4839"
server_key_version: "encryptionkey@0"
I20260620 02:52:27.623150 3974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:27.645146 3974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:27.650063 3974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:27.650395 3974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:27.670575 3974 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40693
I20260620 02:52:27.670621 4040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40693 every 8 connection(s)
I20260620 02:52:27.671983 3974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:52:27.675909 4041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:27.682034 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 3974
I20260620 02:52:27.682284 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:52:27.682933 388 external_mini_cluster.cc:1428] Setting key f10cd47a127868d0ab8a5831754f562b
I20260620 02:52:27.686508 4041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c61e05335fb4e6289bf0f90003f7228: Bootstrap starting.
I20260620 02:52:27.689630 4041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c61e05335fb4e6289bf0f90003f7228: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:27.690821 4041 log.cc:824] T 00000000000000000000000000000000 P 0c61e05335fb4e6289bf0f90003f7228: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:27.693732 4041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c61e05335fb4e6289bf0f90003f7228: No bootstrap required, opened a new log
Jun 20 02:52:27 dist-test-slave-8p8c krb5kdc[3961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:27.701370 4041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c61e05335fb4e6289bf0f90003f7228 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c61e05335fb4e6289bf0f90003f7228" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40693 } }
I20260620 02:52:27.701742 4041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c61e05335fb4e6289bf0f90003f7228 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:27.701841 4041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c61e05335fb4e6289bf0f90003f7228 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c61e05335fb4e6289bf0f90003f7228, State: Initialized, Role: FOLLOWER
I20260620 02:52:27.702558 4041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c61e05335fb4e6289bf0f90003f7228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c61e05335fb4e6289bf0f90003f7228" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40693 } }
I20260620 02:52:27.702800 4041 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c61e05335fb4e6289bf0f90003f7228 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:52:27.702935 4041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c61e05335fb4e6289bf0f90003f7228 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:52:27.703126 4041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c61e05335fb4e6289bf0f90003f7228 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:27.705521 4041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c61e05335fb4e6289bf0f90003f7228 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c61e05335fb4e6289bf0f90003f7228" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40693 } }
I20260620 02:52:27.705775 4044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:27.684944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41728 (local address 127.0.97.62:40693)
0620 02:52:27.685756 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:27.685769 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:27.685810 (+ 41us) server_negotiation.cc:407] Connection header received
0620 02:52:27.687746 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:27.687769 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:27.688285 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:27.688718 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:27.690272 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:27.692216 (+ 1944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:27.693449 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:27.694228 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:27.694453 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:27.698481 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:27.698490 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:27.698505 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:27.698550 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:27.701624 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:27.702424 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:27.702431 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:27.702437 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:27.702595 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:27.702996 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:27.703001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:27.703005 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:27.703598 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:27.703917 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:27.704332 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 02:52:27.704735 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":377,"threads_started":1}
I20260620 02:52:27.706106 4041 leader_election.cc:302] T 00000000000000000000000000000000 P 0c61e05335fb4e6289bf0f90003f7228 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c61e05335fb4e6289bf0f90003f7228; no voters:
I20260620 02:52:27.706835 4041 leader_election.cc:288] T 00000000000000000000000000000000 P 0c61e05335fb4e6289bf0f90003f7228 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:52:27.706946 4046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c61e05335fb4e6289bf0f90003f7228 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:27.707556 4046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c61e05335fb4e6289bf0f90003f7228 [term 1 LEADER]: Becoming Leader. State: Replica: 0c61e05335fb4e6289bf0f90003f7228, State: Running, Role: LEADER
I20260620 02:52:27.708410 4046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c61e05335fb4e6289bf0f90003f7228 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c61e05335fb4e6289bf0f90003f7228" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40693 } }
I20260620 02:52:27.710713 4046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c61e05335fb4e6289bf0f90003f7228 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c61e05335fb4e6289bf0f90003f7228. Latest consensus state: current_term: 1 leader_uuid: "0c61e05335fb4e6289bf0f90003f7228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c61e05335fb4e6289bf0f90003f7228" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40693 } } }
I20260620 02:52:27.711130 4046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c61e05335fb4e6289bf0f90003f7228 [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:27.711550 4048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c61e05335fb4e6289bf0f90003f7228 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c61e05335fb4e6289bf0f90003f7228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c61e05335fb4e6289bf0f90003f7228" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40693 } } }
I20260620 02:52:27.711997 4048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c61e05335fb4e6289bf0f90003f7228 [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:27.714124 4041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c61e05335fb4e6289bf0f90003f7228 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:52:27.714661 4050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:52:27.721396 4050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:52:27.732834 4050 catalog_manager.cc:1347] Generated new cluster ID: d0ac0a1b2cb14effb16bae6e9023f775
I20260620 02:52:27.733040 4050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:52:27.749701 4050 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:52:27.751262 4050 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:52:27.768769 4050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c61e05335fb4e6289bf0f90003f7228: Generated new TSK 0
I20260620 02:52:27.769718 4050 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:52:27.831565 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40693
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:28.213433 4067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:28.213754 4067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:28.213937 4067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:28.221532 4067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:28.221673 4067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:28.221837 4067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:52:28.231757 4067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:28.232993 4067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:28.234474 4067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:28.248486 4073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:28.248571 4075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:28.248610 4072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:28.249236 4067 server_base.cc:1034] running on GCE node
I20260620 02:52:28.249811 4067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:28.250725 4067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:28.251919 4067 hybrid_clock.cc:648] HybridClock initialized: now 1781923948251884 us; error 38 us; skew 500 ppm
I20260620 02:52:28.252236 4067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:28 dist-test-slave-8p8c krb5kdc[3961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:28.255491 4067 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:52:28.256968 4067 webserver.cc:460] Webserver started at http://127.0.97.1:35635/ using document root <none> and password file <none>
I20260620 02:52:28.257673 4067 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:28.257817 4067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:28.258123 4067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:28.260915 4067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-0/data/instance:
uuid: "02295a4828c54ec182807d18fb32dfc0"
format_stamp: "Formatted at 2026-06-20 02:52:28 on dist-test-slave-8p8c"
server_key: "09218b14e1cb3dadbea0a7c2b741936b"
server_key_iv: "559390516de46514724fc022613596b6"
server_key_version: "encryptionkey@0"
I20260620 02:52:28.261768 4067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-0/wal/instance:
uuid: "02295a4828c54ec182807d18fb32dfc0"
format_stamp: "Formatted at 2026-06-20 02:52:28 on dist-test-slave-8p8c"
server_key: "09218b14e1cb3dadbea0a7c2b741936b"
server_key_iv: "559390516de46514724fc022613596b6"
server_key_version: "encryptionkey@0"
I20260620 02:52:28.266853 4067 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 02:52:28.270237 4082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:28.271584 4067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:52:28.271752 4067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-0/wal
uuid: "02295a4828c54ec182807d18fb32dfc0"
format_stamp: "Formatted at 2026-06-20 02:52:28 on dist-test-slave-8p8c"
server_key: "09218b14e1cb3dadbea0a7c2b741936b"
server_key_iv: "559390516de46514724fc022613596b6"
server_key_version: "encryptionkey@0"
I20260620 02:52:28.271950 4067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:28.290009 4067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:28.294226 4067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:28.294783 4067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:28.295918 4067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:52:28.297372 4067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:28.297485 4067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:28.297668 4067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:28.297767 4067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:28.333415 4067 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35217
I20260620 02:52:28.333462 4194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35217 every 8 connection(s)
I20260620 02:52:28.334678 4067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:52:28.335794 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 4067
I20260620 02:52:28.336014 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:52:28.336796 388 external_mini_cluster.cc:1428] Setting key 230ba13ecbe11787948a8de89d6bb941
Jun 20 02:52:28 dist-test-slave-8p8c krb5kdc[3961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:52:28.357435 4197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:28.337688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52945 (local address 127.0.97.62:40693)
0620 02:52:28.338187 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:28.338193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:28.339330 (+ 1137us) server_negotiation.cc:407] Connection header received
0620 02:52:28.341078 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:28.341083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:28.341164 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:28.341294 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:28.342909 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:28.343917 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:28.345577 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:28.346342 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:28.346498 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:28.350027 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:28.350036 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:28.350043 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:28.350096 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:28.353929 (+ 3833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:28.355062 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:28.355070 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:28.355075 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:28.355221 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:28.355713 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:28.355717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:28.355719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:28.356095 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:28.356215 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:28.357006 (+ 791us) server_negotiation.cc:299] Negotiation successful
0620 02:52:28.357176 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":253,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 02:52:28.358700 4198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:28.338417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40693 (local address 127.0.97.1:52945)
0620 02:52:28.339183 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:28.339225 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:28.340820 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:28.341533 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:28.341542 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:28.342087 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:28.342746 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:28.342765 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:28.344093 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:28.344098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:28.345422 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:28.345435 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:28.346587 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:28.346603 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:28.346812 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:28.347556 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:28.347582 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:28.349802 (+ 2220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:28.354111 (+ 4309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:28.354119 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:28.354134 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:28.354910 (+ 776us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:28.355353 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:28.355357 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:28.355359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:28.355571 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:28.356235 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:28.356241 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:28.356589 (+ 348us) client_negotiation.cc:769] Sending connection context
0620 02:52:28.356830 (+ 241us) client_negotiation.cc:240] Negotiation successful
0620 02:52:28.357307 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":317,"threads_started":1}
I20260620 02:52:28.360519 4195 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40693
I20260620 02:52:28.360841 4195 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:28.361743 4195 heartbeater.cc:511] Master 127.0.97.62:40693 requested a full tablet report, sending...
I20260620 02:52:28.364353 4006 ts_manager.cc:194] Registered new tserver with Master: 02295a4828c54ec182807d18fb32dfc0 (127.0.97.1:35217)
I20260620 02:52:28.366578 4006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52945
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:52:28.385229 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40693
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:28.709824 4201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:28.710115 4201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:28.710276 4201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:28.717684 4201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:28.717801 4201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:28.717983 4201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:52:28.729074 4201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:28.730418 4201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:28.731763 4201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:28.745543 4209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:28.745888 4206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:28.746115 4207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:28.747146 4201 server_base.cc:1034] running on GCE node
I20260620 02:52:28.747877 4201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:28.749122 4201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:28.750474 4201 hybrid_clock.cc:648] HybridClock initialized: now 1781923948750428 us; error 62 us; skew 500 ppm
I20260620 02:52:28.750854 4201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:28 dist-test-slave-8p8c krb5kdc[3961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:28.754626 4201 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:52:28.756289 4201 webserver.cc:460] Webserver started at http://127.0.97.2:35539/ using document root <none> and password file <none>
I20260620 02:52:28.757019 4201 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:28.757166 4201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:28.757476 4201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:28.760479 4201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-1/data/instance:
uuid: "7ec5d955dcc74396908182ebd9de89b0"
format_stamp: "Formatted at 2026-06-20 02:52:28 on dist-test-slave-8p8c"
server_key: "35ff7e4f287517cff26d74ca83691aab"
server_key_iv: "5a8c43738ca054bd9d8456d1d0f57d6b"
server_key_version: "encryptionkey@0"
I20260620 02:52:28.761282 4201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7ec5d955dcc74396908182ebd9de89b0"
format_stamp: "Formatted at 2026-06-20 02:52:28 on dist-test-slave-8p8c"
server_key: "35ff7e4f287517cff26d74ca83691aab"
server_key_iv: "5a8c43738ca054bd9d8456d1d0f57d6b"
server_key_version: "encryptionkey@0"
I20260620 02:52:28.767266 4201 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260620 02:52:28.771425 4216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:28.773136 4201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 02:52:28.773309 4201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-1/wal
uuid: "7ec5d955dcc74396908182ebd9de89b0"
format_stamp: "Formatted at 2026-06-20 02:52:28 on dist-test-slave-8p8c"
server_key: "35ff7e4f287517cff26d74ca83691aab"
server_key_iv: "5a8c43738ca054bd9d8456d1d0f57d6b"
server_key_version: "encryptionkey@0"
I20260620 02:52:28.773583 4201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:28.792368 4201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:28.797482 4201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:28.797816 4201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:28.798959 4201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:52:28.800401 4201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:28.800511 4201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:28.800673 4201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:28.800745 4201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:28.835844 4201 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37331
I20260620 02:52:28.835933 4328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37331 every 8 connection(s)
I20260620 02:52:28.837126 4201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:52:28.844190 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 4201
I20260620 02:52:28.844403 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:52:28.844931 388 external_mini_cluster.cc:1428] Setting key 1fd55465025f3de5d8475ee0a9433081
Jun 20 02:52:28 dist-test-slave-8p8c krb5kdc[3961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:28.861143 4197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:28.840063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60417 (local address 127.0.97.62:40693)
0620 02:52:28.840304 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:28.840311 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:28.841604 (+ 1293us) server_negotiation.cc:407] Connection header received
0620 02:52:28.843453 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:28.843458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:28.843531 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:28.843633 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:28.845707 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:28.846825 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:28.849488 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:28.849970 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:28.850083 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:28.854548 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:28.854553 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:28.854557 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:28.854593 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:28.857290 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:28.858270 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:28.858274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:28.858277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:28.858474 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:28.858952 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:28.858956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:28.858958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:28.859315 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:28.859472 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:28.860392 (+ 920us) server_negotiation.cc:299] Negotiation successful
0620 02:52:28.860660 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 02:52:28.862134 4331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:28.840624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40693 (local address 127.0.97.2:60417)
0620 02:52:28.841425 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:28.841471 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:28.843165 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:28.843873 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:28.843887 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:28.844713 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:28.845524 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:28.845544 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:28.847008 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:28.847012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:28.849269 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:28.849301 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:28.850246 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:28.850256 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:28.850614 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:28.851400 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:28.851442 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:28.854239 (+ 2797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:28.857441 (+ 3202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:28.857453 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:28.857474 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:28.858145 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:28.858591 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:28.858598 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:28.858605 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:28.858844 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:28.859469 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:28.859481 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:28.859967 (+ 486us) client_negotiation.cc:769] Sending connection context
0620 02:52:28.860294 (+ 327us) client_negotiation.cc:240] Negotiation successful
0620 02:52:28.860905 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":365,"threads_started":1}
I20260620 02:52:28.863984 4329 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40693
I20260620 02:52:28.864357 4329 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:28.865236 4329 heartbeater.cc:511] Master 127.0.97.62:40693 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:52:28.867007 4006 ts_manager.cc:194] Registered new tserver with Master: 7ec5d955dcc74396908182ebd9de89b0 (127.0.97.2:37331)
I20260620 02:52:28.867838 4006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60417
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:52:28.895198 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40693
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:29.231326 4334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:29.231722 4334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:29.231891 4334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:29.239289 4334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:29.239449 4334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:29.239642 4334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:52:29.249464 4334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:29.250844 4334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:29.252179 4334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:29.265851 4339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:29.265875 4340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:29.265828 4342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:29.266778 4334 server_base.cc:1034] running on GCE node
I20260620 02:52:29.267319 4334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:29.268229 4334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:29.269475 4334 hybrid_clock.cc:648] HybridClock initialized: now 1781923949269413 us; error 46 us; skew 500 ppm
I20260620 02:52:29.269836 4334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:29 dist-test-slave-8p8c krb5kdc[3961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:29.273506 4334 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:52:29.275166 4334 webserver.cc:460] Webserver started at http://127.0.97.3:38661/ using document root <none> and password file <none>
I20260620 02:52:29.275985 4334 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:29.276161 4334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:29.276507 4334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:29.279789 4334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-2/data/instance:
uuid: "21c42f40dedb49338cc7d64703e886f7"
format_stamp: "Formatted at 2026-06-20 02:52:29 on dist-test-slave-8p8c"
server_key: "ef5f35f9a6b8ae58e440997bbf4bd12c"
server_key_iv: "106ed597ce2e7682458cef5c7429831a"
server_key_version: "encryptionkey@0"
I20260620 02:52:29.280592 4334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-2/wal/instance:
uuid: "21c42f40dedb49338cc7d64703e886f7"
format_stamp: "Formatted at 2026-06-20 02:52:29 on dist-test-slave-8p8c"
server_key: "ef5f35f9a6b8ae58e440997bbf4bd12c"
server_key_iv: "106ed597ce2e7682458cef5c7429831a"
server_key_version: "encryptionkey@0"
I20260620 02:52:29.285837 4334 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:52:29.289702 4349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:29.291255 4334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:52:29.291445 4334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-2/wal
uuid: "21c42f40dedb49338cc7d64703e886f7"
format_stamp: "Formatted at 2026-06-20 02:52:29 on dist-test-slave-8p8c"
server_key: "ef5f35f9a6b8ae58e440997bbf4bd12c"
server_key_iv: "106ed597ce2e7682458cef5c7429831a"
server_key_version: "encryptionkey@0"
I20260620 02:52:29.291672 4334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:29.317323 4334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:29.322077 4334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:29.322397 4334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:29.323400 4334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:52:29.324761 4334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:29.324832 4334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:29.324961 4334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:29.325011 4334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:29.361199 4334 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34273
I20260620 02:52:29.361287 4461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34273 every 8 connection(s)
I20260620 02:52:29.362597 4334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:52:29.364990 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 4334
I20260620 02:52:29.365198 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781923914418765-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:52:29.365664 388 external_mini_cluster.cc:1428] Setting key c5751fd38c928472ce6ab3519561fb06
I20260620 02:52:29.369992 4195 heartbeater.cc:503] Master 127.0.97.62:40693 was elected leader, sending a full tablet report...
Jun 20 02:52:29 dist-test-slave-8p8c krb5kdc[3961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:29.385046 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:29.366264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35215 (local address 127.0.97.62:40693)
0620 02:52:29.366821 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:29.366827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:29.367238 (+ 411us) server_negotiation.cc:407] Connection header received
0620 02:52:29.369425 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:29.369429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:29.369499 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:29.369605 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:29.371829 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:29.372607 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:29.374184 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:29.374676 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:29.374777 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:29.378654 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:29.378659 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:29.378662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:29.378693 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:29.381942 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:29.382823 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:29.382828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:29.382830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:29.383022 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:29.383484 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:29.383488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:29.383491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:29.383767 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:29.383880 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:29.384609 (+ 729us) server_negotiation.cc:299] Negotiation successful
0620 02:52:29.384776 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":329,"threads_started":1}
I20260620 02:52:29.386044 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:29.365841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40693 (local address 127.0.97.3:35215)
0620 02:52:29.366775 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:29.366827 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:29.369090 (+ 2263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:29.369949 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:29.369961 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:29.370897 (+ 936us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:29.371655 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:29.371683 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:29.372750 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:29.372753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:29.374045 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:29.374057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:29.374956 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:29.374963 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:29.375198 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:29.375933 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:29.375955 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:29.378132 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:29.382090 (+ 3958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:29.382097 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:29.382112 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:29.382668 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:29.383161 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:29.383164 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:29.383167 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:29.383349 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:29.383925 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:29.383931 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:29.384248 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 02:52:29.384480 (+ 232us) client_negotiation.cc:240] Negotiation successful
0620 02:52:29.384882 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":347,"threads_started":1}
I20260620 02:52:29.387727 4462 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40693
I20260620 02:52:29.388065 4462 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:29.388739 4462 heartbeater.cc:511] Master 127.0.97.62:40693 requested a full tablet report, sending...
I20260620 02:52:29.390314 4006 ts_manager.cc:194] Registered new tserver with Master: 21c42f40dedb49338cc7d64703e886f7 (127.0.97.3:34273)
I20260620 02:52:29.391176 4006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35215
I20260620 02:52:29.394212 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 02:52:29 dist-test-slave-8p8c krb5kdc[3961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:52:29 dist-test-slave-8p8c krb5kdc[3961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:52:29.444667 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:29.429131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39108 (local address 127.0.97.1:35217)
0620 02:52:29.429855 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:29.429862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:29.429888 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:52:29.429946 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:29.429952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:29.430179 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:29.430352 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:29.431763 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:29.432741 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:29.433965 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:29.434613 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:29.434746 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:29.438912 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:29.438924 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:29.438940 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:29.438978 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:29.441801 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:29.442573 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:29.442581 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:29.442584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:29.442708 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:29.443098 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:29.443107 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:29.443109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:29.443431 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:29.443756 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:29.444048 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 02:52:29.444317 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":345,"threads_started":1}
Jun 20 02:52:29 dist-test-slave-8p8c krb5kdc[3961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:29.463681 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:29.449457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41732 (local address 127.0.97.62:40693)
0620 02:52:29.449714 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:29.449723 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:29.449833 (+ 110us) server_negotiation.cc:407] Connection header received
0620 02:52:29.450052 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:29.450056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:29.450128 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:29.450239 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:29.451661 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:29.452398 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:29.453656 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:29.454063 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:29.454200 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:29.458283 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:29.458290 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:29.458293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:29.458333 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:29.461046 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:29.461696 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:29.461701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:29.461704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:29.461833 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:29.462208 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:29.462211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:29.462213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:29.462607 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:29.462792 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:29.463153 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 02:52:29.463319 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260620 02:52:29.464318 4006 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41732
I20260620 02:52:29.465399 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 4067
I20260620 02:52:29.479362 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:29.466263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39120 (local address 127.0.97.1:35217)
0620 02:52:29.466519 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:29.466529 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:29.466868 (+ 339us) server_negotiation.cc:407] Connection header received
0620 02:52:29.466998 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:29.467002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:29.467070 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:29.467155 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:29.468360 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:29.469058 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:29.470632 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:29.471120 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:29.471225 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:29.473536 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:29.473542 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:29.473545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:29.473580 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:29.476396 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:29.477304 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:29.477312 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:29.477316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:29.477467 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:29.477850 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:29.477854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:29.477857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:29.478196 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:29.478498 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:29.478807 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 02:52:29.478979 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 02:52:29.479993 4190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:29.563578 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 4067
I20260620 02:52:29.579847 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 4201
Jun 20 02:52:29 dist-test-slave-8p8c krb5kdc[3961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:52:29.595829 4475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:29.580815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49710 (local address 127.0.97.2:37331)
0620 02:52:29.581389 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:29.581398 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:29.581425 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:52:29.581491 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:29.581498 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:29.581707 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:29.581842 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:29.582992 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:29.584010 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:29.585189 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:29.585731 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:29.585832 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:29.590011 (+ 4179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:29.590023 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:29.590061 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:29.590094 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:29.593107 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:29.593800 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:29.593808 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:29.593810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:29.593949 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:29.594258 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:29.594262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:29.594264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:29.594684 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:29.594940 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:29.595188 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 02:52:29.595373 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":308,"threads_started":1}
I20260620 02:52:29.597237 4324 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:29.685400 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 4201
I20260620 02:52:29.701489 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 4334
Jun 20 02:52:29 dist-test-slave-8p8c krb5kdc[3961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:52:29.718523 4477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:29.702791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59664 (local address 127.0.97.3:34273)
0620 02:52:29.703462 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:29.703469 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:29.703498 (+ 29us) server_negotiation.cc:407] Connection header received
0620 02:52:29.703558 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:29.703564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:29.703780 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:29.703907 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:29.705151 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:29.706187 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:29.707583 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:29.708144 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:29.708250 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:29.712483 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:29.712496 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:29.712513 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:29.712551 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:29.715655 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:29.716314 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:29.716322 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:29.716324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:29.716439 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:29.716797 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:29.716806 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:29.716808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:29.717141 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:29.717455 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:29.717745 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 02:52:29.718036 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":360,"threads_started":1}
I20260620 02:52:29.719856 4457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:29.807292 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 4334
I20260620 02:52:29.823019 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 3974
I20260620 02:52:29.824252 4036 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:29.926838 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 3974
2026-06-20T02:52:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3997 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:52:30 dist-test-slave-8p8c krb5kdc[4482](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:52:30 dist-test-slave-8p8c krb5kdc[4482](info): set up 2 sockets
Jun 20 02:52:30 dist-test-slave-8p8c krb5kdc[4482](info): commencing operation
krb5kdc: starting...
W20260620 02:52:31.055071 388 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:52:31 dist-test-slave-8p8c krb5kdc[4482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:52:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:52:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:52:31.262238 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41605
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:31.599028 4495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:31.599363 4495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:31.599519 4495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:31.606642 4495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:52:31.606778 4495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:31.606875 4495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:31.606976 4495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:52:31.607052 4495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:52:31.616672 4495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:31.617969 4495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:31.619285 4495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:31.630927 4503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:31.630970 4500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:31.631060 4501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:31.632206 4495 server_base.cc:1034] running on GCE node
I20260620 02:52:31.633020 4495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:31.634539 4495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:31.635784 4495 hybrid_clock.cc:648] HybridClock initialized: now 1781923951635732 us; error 47 us; skew 500 ppm
I20260620 02:52:31.636145 4495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:31 dist-test-slave-8p8c krb5kdc[4482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:31.639575 4495 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:52:31.640956 4495 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:31.641098 4495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:31.641443 4495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:31.644681 4495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "8e0c374527c340c78eec39261b2cff36"
format_stamp: "Formatted at 2026-06-20 02:52:31 on dist-test-slave-8p8c"
server_key: "d968565cb117d75ce98476563046fd81"
server_key_iv: "e93998bce1392b10b5bdd31abd5988cd"
server_key_version: "encryptionkey@0"
I20260620 02:52:31.645592 4495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "8e0c374527c340c78eec39261b2cff36"
format_stamp: "Formatted at 2026-06-20 02:52:31 on dist-test-slave-8p8c"
server_key: "d968565cb117d75ce98476563046fd81"
server_key_iv: "e93998bce1392b10b5bdd31abd5988cd"
server_key_version: "encryptionkey@0"
I20260620 02:52:31.651054 4495 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 02:52:31.654760 4509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:31.656222 4495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:52:31.656445 4495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "8e0c374527c340c78eec39261b2cff36"
format_stamp: "Formatted at 2026-06-20 02:52:31 on dist-test-slave-8p8c"
server_key: "d968565cb117d75ce98476563046fd81"
server_key_iv: "e93998bce1392b10b5bdd31abd5988cd"
server_key_version: "encryptionkey@0"
I20260620 02:52:31.656685 4495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:31.676878 4495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:31.681519 4495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:31.681816 4495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:31.701337 4495 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41605
I20260620 02:52:31.701356 4560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41605 every 8 connection(s)
I20260620 02:52:31.702275 4495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:52:31.706033 4561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:31.712361 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 4495
I20260620 02:52:31.712626 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:52:31.713490 388 external_mini_cluster.cc:1428] Setting key f3427c769b3dfd76c3ae5c7c1a6cd7ab
I20260620 02:52:31.718847 4561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e0c374527c340c78eec39261b2cff36: Bootstrap starting.
I20260620 02:52:31.722036 4561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e0c374527c340c78eec39261b2cff36: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:31.723197 4561 log.cc:824] T 00000000000000000000000000000000 P 8e0c374527c340c78eec39261b2cff36: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:31.726419 4561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e0c374527c340c78eec39261b2cff36: No bootstrap required, opened a new log
Jun 20 02:52:31 dist-test-slave-8p8c krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:31.737672 4561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e0c374527c340c78eec39261b2cff36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e0c374527c340c78eec39261b2cff36" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41605 } }
I20260620 02:52:31.738153 4561 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e0c374527c340c78eec39261b2cff36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:31.738292 4561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e0c374527c340c78eec39261b2cff36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0c374527c340c78eec39261b2cff36, State: Initialized, Role: FOLLOWER
I20260620 02:52:31.739035 4561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e0c374527c340c78eec39261b2cff36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e0c374527c340c78eec39261b2cff36" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41605 } }
I20260620 02:52:31.739252 4564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:31.715939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51680 (local address 127.0.97.62:41605)
0620 02:52:31.716772 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:31.716785 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:31.716831 (+ 46us) server_negotiation.cc:407] Connection header received
0620 02:52:31.718618 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:31.718642 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:31.719097 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:31.719487 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:31.720938 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:31.722307 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:31.723638 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:31.724354 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:31.724540 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:31.729386 (+ 4846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:31.729397 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:31.729414 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:31.729459 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:31.734337 (+ 4878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:31.735331 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:31.735338 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:31.735345 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:31.735541 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:31.736013 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:31.736017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:31.736020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:31.736638 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:31.736851 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:31.737420 (+ 569us) server_negotiation.cc:299] Negotiation successful
0620 02:52:31.737847 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":360,"threads_started":1}
I20260620 02:52:31.739303 4561 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e0c374527c340c78eec39261b2cff36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:52:31.739450 4561 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e0c374527c340c78eec39261b2cff36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:52:31.739653 4561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e0c374527c340c78eec39261b2cff36 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:31.742218 4561 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e0c374527c340c78eec39261b2cff36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e0c374527c340c78eec39261b2cff36" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41605 } }
I20260620 02:52:31.742812 4561 leader_election.cc:302] T 00000000000000000000000000000000 P 8e0c374527c340c78eec39261b2cff36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e0c374527c340c78eec39261b2cff36; no voters:
I20260620 02:52:31.743691 4561 leader_election.cc:288] T 00000000000000000000000000000000 P 8e0c374527c340c78eec39261b2cff36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:52:31.743840 4566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e0c374527c340c78eec39261b2cff36 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:31.744588 4566 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e0c374527c340c78eec39261b2cff36 [term 1 LEADER]: Becoming Leader. State: Replica: 8e0c374527c340c78eec39261b2cff36, State: Running, Role: LEADER
I20260620 02:52:31.745302 4566 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e0c374527c340c78eec39261b2cff36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e0c374527c340c78eec39261b2cff36" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41605 } }
I20260620 02:52:31.747073 4561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e0c374527c340c78eec39261b2cff36 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:52:31.748939 4568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e0c374527c340c78eec39261b2cff36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e0c374527c340c78eec39261b2cff36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e0c374527c340c78eec39261b2cff36" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41605 } } }
I20260620 02:52:31.749439 4568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e0c374527c340c78eec39261b2cff36 [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:31.751071 4567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e0c374527c340c78eec39261b2cff36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e0c374527c340c78eec39261b2cff36. Latest consensus state: current_term: 1 leader_uuid: "8e0c374527c340c78eec39261b2cff36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e0c374527c340c78eec39261b2cff36" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41605 } } }
I20260620 02:52:31.751416 4567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e0c374527c340c78eec39261b2cff36 [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:31.752600 4573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:52:31.758065 4573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:52:31.768168 4573 catalog_manager.cc:1347] Generated new cluster ID: a7531e0b9c1940f4b6e50fca8b5a8ee1
I20260620 02:52:31.768289 4573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:52:31.795845 4573 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:52:31.797775 4573 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:52:31.812700 4573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e0c374527c340c78eec39261b2cff36: Generated new TSK 0
I20260620 02:52:31.813861 4573 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:52:31.872227 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41605
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:32.202823 4587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:32.203184 4587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:32.203347 4587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:32.210397 4587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:32.210551 4587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:32.210734 4587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:52:32.220577 4587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:32.221922 4587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:32.223297 4587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:32.236441 4592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:32.236398 4593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:32.236362 4595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:32.236934 4587 server_base.cc:1034] running on GCE node
I20260620 02:52:32.237572 4587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:32.238823 4587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:32.240078 4587 hybrid_clock.cc:648] HybridClock initialized: now 1781923952240017 us; error 55 us; skew 500 ppm
I20260620 02:52:32.240437 4587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:32 dist-test-slave-8p8c krb5kdc[4482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:32.243700 4587 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:52:32.244839 4587 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:32.244925 4587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:32.245214 4587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:32.248355 4587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-0/data/instance:
uuid: "802b696d7e784ff680dca6822c264168"
format_stamp: "Formatted at 2026-06-20 02:52:32 on dist-test-slave-8p8c"
server_key: "8ad87661c2a532dc49a3bbd68f1464ae"
server_key_iv: "a797c654c9e814186aeb98aa35fcf140"
server_key_version: "encryptionkey@0"
I20260620 02:52:32.249153 4587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-0/wal/instance:
uuid: "802b696d7e784ff680dca6822c264168"
format_stamp: "Formatted at 2026-06-20 02:52:32 on dist-test-slave-8p8c"
server_key: "8ad87661c2a532dc49a3bbd68f1464ae"
server_key_iv: "a797c654c9e814186aeb98aa35fcf140"
server_key_version: "encryptionkey@0"
I20260620 02:52:32.253940 4587 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 02:52:32.257637 4601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:32.259069 4587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:52:32.259222 4587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-0/wal
uuid: "802b696d7e784ff680dca6822c264168"
format_stamp: "Formatted at 2026-06-20 02:52:32 on dist-test-slave-8p8c"
server_key: "8ad87661c2a532dc49a3bbd68f1464ae"
server_key_iv: "a797c654c9e814186aeb98aa35fcf140"
server_key_version: "encryptionkey@0"
I20260620 02:52:32.259419 4587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:32.287561 4587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:32.291992 4587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:32.292280 4587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:32.293334 4587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:52:32.294759 4587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:32.294843 4587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:32.294986 4587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:32.295070 4587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:32.329573 4587 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46473
I20260620 02:52:32.329676 4713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46473 every 8 connection(s)
I20260620 02:52:32.330538 4587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:52:32.333739 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 4587
I20260620 02:52:32.334028 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:52:32.334769 388 external_mini_cluster.cc:1428] Setting key a0f25c4be88f18f6638991fca53e4e84
Jun 20 02:52:32 dist-test-slave-8p8c krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:32.355892 4716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:32.333410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43747 (local address 127.0.97.62:41605)
0620 02:52:32.333913 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:32.333921 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:32.335129 (+ 1208us) server_negotiation.cc:407] Connection header received
0620 02:52:32.337213 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:32.337220 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:32.337309 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:32.337476 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:32.339007 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:32.340256 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:32.342000 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:32.342648 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:32.342761 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:32.347435 (+ 4674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:32.347445 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:32.347454 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:32.347503 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:32.351706 (+ 4203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:32.352676 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:32.352683 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:32.352687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:32.352852 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:32.353405 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:32.353422 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:32.353429 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:32.353898 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:32.354123 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:32.355030 (+ 907us) server_negotiation.cc:299] Negotiation successful
0620 02:52:32.355406 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":232,"threads_started":1}
I20260620 02:52:32.356549 4717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:32.334058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41605 (local address 127.0.97.1:43747)
0620 02:52:32.334945 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:32.335003 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:32.336953 (+ 1950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:32.337665 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:32.337673 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:32.338163 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:32.338821 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:32.338839 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:32.340457 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:32.340461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:32.341778 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:32.341791 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:32.342907 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:32.342916 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:32.343259 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:32.344065 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:32.344108 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:32.347150 (+ 3042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:32.351885 (+ 4735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:32.351892 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:32.351907 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:32.352516 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:32.353009 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:32.353012 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:32.353015 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:32.353211 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:32.354146 (+ 935us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:32.354155 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:32.354565 (+ 410us) client_negotiation.cc:769] Sending connection context
0620 02:52:32.354834 (+ 269us) client_negotiation.cc:240] Negotiation successful
0620 02:52:32.355373 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":389,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:52:32.358155 4714 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41605
I20260620 02:52:32.358560 4714 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:32.359316 4714 heartbeater.cc:511] Master 127.0.97.62:41605 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 02:52:32.361913 4526 ts_manager.cc:194] Registered new tserver with Master: 802b696d7e784ff680dca6822c264168 (127.0.97.1:46473)
I20260620 02:52:32.364301 4526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43747
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:52:32.386533 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41605
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:32.717984 4720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:32.718253 4720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:32.718348 4720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:32.725589 4720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:32.725695 4720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:32.725808 4720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:52:32.736097 4720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:32.737995 4720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:32.739668 4720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:32.754004 4725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:32.753996 4726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:32.753996 4728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:32.754626 4720 server_base.cc:1034] running on GCE node
I20260620 02:52:32.755362 4720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:32.756390 4720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:32.757659 4720 hybrid_clock.cc:648] HybridClock initialized: now 1781923952757606 us; error 44 us; skew 500 ppm
I20260620 02:52:32.758037 4720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:32 dist-test-slave-8p8c krb5kdc[4482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:32.767717 4720 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:52:32.769088 4720 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:32.769243 4720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:32.769598 4720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:32.772706 4720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-1/data/instance:
uuid: "cae0a74e639e4bc988864f8f8e5951f1"
format_stamp: "Formatted at 2026-06-20 02:52:32 on dist-test-slave-8p8c"
server_key: "6d23480add08f442ddb24f27eef977aa"
server_key_iv: "cdd920a1813c386eb4a23548c96dd5de"
server_key_version: "encryptionkey@0"
I20260620 02:52:32.773638 4720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-1/wal/instance:
uuid: "cae0a74e639e4bc988864f8f8e5951f1"
format_stamp: "Formatted at 2026-06-20 02:52:32 on dist-test-slave-8p8c"
server_key: "6d23480add08f442ddb24f27eef977aa"
server_key_iv: "cdd920a1813c386eb4a23548c96dd5de"
server_key_version: "encryptionkey@0"
I20260620 02:52:32.779583 4720 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260620 02:52:32.783306 4734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:32.784775 4720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:52:32.785022 4720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-1/wal
uuid: "cae0a74e639e4bc988864f8f8e5951f1"
format_stamp: "Formatted at 2026-06-20 02:52:32 on dist-test-slave-8p8c"
server_key: "6d23480add08f442ddb24f27eef977aa"
server_key_iv: "cdd920a1813c386eb4a23548c96dd5de"
server_key_version: "encryptionkey@0"
I20260620 02:52:32.785282 4720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:32.816385 4720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:32.821202 4720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:32.821480 4720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:32.822476 4720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:52:32.823791 4720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:32.823905 4720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:32.823994 4720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:32.824049 4720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:32.858695 4720 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44357
I20260620 02:52:32.858777 4846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44357 every 8 connection(s)
I20260620 02:52:32.859540 4720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:52:32.867146 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 4720
I20260620 02:52:32.867367 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:52:32.867893 388 external_mini_cluster.cc:1428] Setting key 47096220f722de68f798650dc4d35d80
Jun 20 02:52:32 dist-test-slave-8p8c krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:32.880193 4849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:32.862357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52231 (local address 127.0.97.62:41605)
0620 02:52:32.862859 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:32.862864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:32.863990 (+ 1126us) server_negotiation.cc:407] Connection header received
0620 02:52:32.865869 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:32.865881 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:32.865974 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:32.866100 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:32.867877 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:32.868654 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:32.870192 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:32.870704 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:32.870835 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:32.874329 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:32.874341 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:32.874344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:32.874424 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:32.877099 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:32.877842 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:32.877847 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:32.877851 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:32.877979 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:32.878459 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:32.878464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:32.878468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:32.878733 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:32.878896 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:32.879583 (+ 687us) server_negotiation.cc:299] Negotiation successful
0620 02:52:32.879798 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":261,"threads_started":1}
I20260620 02:52:32.881207 4850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:32.863030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41605 (local address 127.0.97.2:52231)
0620 02:52:32.863821 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:32.863865 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:32.865600 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:32.866461 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:32.866470 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:32.867009 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:32.867703 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:32.867723 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:32.868804 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:32.868809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:32.870052 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:32.870063 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:32.870930 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:32.870937 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:32.871157 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:32.871865 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:32.871892 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:32.874089 (+ 2197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:32.877241 (+ 3152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:32.877251 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:32.877270 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:32.877709 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:32.878103 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:32.878108 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:32.878112 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:32.878306 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:32.878871 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:32.878879 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:32.879232 (+ 353us) client_negotiation.cc:769] Sending connection context
0620 02:52:32.879507 (+ 275us) client_negotiation.cc:240] Negotiation successful
0620 02:52:32.880062 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":360,"threads_started":1}
I20260620 02:52:32.882905 4847 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41605
I20260620 02:52:32.883279 4847 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:32.884262 4847 heartbeater.cc:511] Master 127.0.97.62:41605 requested a full tablet report, sending...
I20260620 02:52:32.886199 4526 ts_manager.cc:194] Registered new tserver with Master: cae0a74e639e4bc988864f8f8e5951f1 (127.0.97.2:44357)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:52:32.887616 4526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52231
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:52:32.924567 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41605
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:33.252118 4853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:33.252434 4853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:33.252605 4853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:33.259716 4853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:33.259862 4853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:33.260039 4853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:52:33.269691 4853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:33.270984 4853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:33.272290 4853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:33.285537 4858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:33.285670 4861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:33.285686 4859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:33.286882 4853 server_base.cc:1034] running on GCE node
I20260620 02:52:33.287535 4853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:33.288551 4853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:33.289798 4853 hybrid_clock.cc:648] HybridClock initialized: now 1781923953289737 us; error 43 us; skew 500 ppm
I20260620 02:52:33.290194 4853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:33 dist-test-slave-8p8c krb5kdc[4482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:33.293545 4853 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:52:33.294723 4853 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:33.294813 4853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:33.295101 4853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:33.298029 4853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-2/data/instance:
uuid: "97aae3a894dd487bba5beddf22e9e261"
format_stamp: "Formatted at 2026-06-20 02:52:33 on dist-test-slave-8p8c"
server_key: "99dc13ec5af11d2c2499e299aee2507d"
server_key_iv: "8d99af841ede2b759d5dd656138dfc7f"
server_key_version: "encryptionkey@0"
I20260620 02:52:33.298884 4853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-2/wal/instance:
uuid: "97aae3a894dd487bba5beddf22e9e261"
format_stamp: "Formatted at 2026-06-20 02:52:33 on dist-test-slave-8p8c"
server_key: "99dc13ec5af11d2c2499e299aee2507d"
server_key_iv: "8d99af841ede2b759d5dd656138dfc7f"
server_key_version: "encryptionkey@0"
I20260620 02:52:33.303975 4853 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 02:52:33.307366 4867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:33.308691 4853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:52:33.308871 4853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-2/wal
uuid: "97aae3a894dd487bba5beddf22e9e261"
format_stamp: "Formatted at 2026-06-20 02:52:33 on dist-test-slave-8p8c"
server_key: "99dc13ec5af11d2c2499e299aee2507d"
server_key_iv: "8d99af841ede2b759d5dd656138dfc7f"
server_key_version: "encryptionkey@0"
I20260620 02:52:33.309093 4853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:33.334944 4853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:33.339514 4853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:33.339745 4853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:33.340725 4853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:52:33.342056 4853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:33.342146 4853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:33.342221 4853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:33.342279 4853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:33.368232 4714 heartbeater.cc:503] Master 127.0.97.62:41605 was elected leader, sending a full tablet report...
I20260620 02:52:33.378532 4853 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39669
I20260620 02:52:33.378580 4979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39669 every 8 connection(s)
I20260620 02:52:33.379434 4853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:52:33.384627 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 4853
I20260620 02:52:33.384933 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:52:33.385411 388 external_mini_cluster.cc:1428] Setting key b3f639c670db37060eb3c8b384c87a57
Jun 20 02:52:33 dist-test-slave-8p8c krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:33.400931 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:33.382077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48793 (local address 127.0.97.62:41605)
0620 02:52:33.382622 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:33.382627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:33.383643 (+ 1016us) server_negotiation.cc:407] Connection header received
0620 02:52:33.385818 (+ 2175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:33.385825 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:33.385914 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:33.386063 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:33.387813 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:33.388738 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:33.390643 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:33.391101 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:33.391246 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:33.394975 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:33.394980 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:33.394984 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:33.395023 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:33.397710 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:33.398537 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:33.398542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:33.398544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:33.398662 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:33.399104 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:33.399116 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:33.399120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:33.399389 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:33.399551 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:33.400276 (+ 725us) server_negotiation.cc:299] Negotiation successful
0620 02:52:33.400509 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":311,"threads_started":1}
I20260620 02:52:33.401916 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:33.382747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41605 (local address 127.0.97.3:48793)
0620 02:52:33.383468 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:33.383517 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:33.385501 (+ 1984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:33.386308 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:33.386342 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:33.387026 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:33.387623 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:33.387640 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:33.388890 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:33.388897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:33.390493 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:33.390507 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:33.391331 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:33.391338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:33.391583 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:33.392301 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:33.392324 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:33.394726 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:33.397856 (+ 3130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:33.397866 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:33.397885 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:33.398330 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:33.398779 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:33.398785 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:33.398789 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:33.398979 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:33.399538 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:33.399546 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:33.399886 (+ 340us) client_negotiation.cc:769] Sending connection context
0620 02:52:33.400160 (+ 274us) client_negotiation.cc:240] Negotiation successful
0620 02:52:33.400738 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":300,"threads_started":1}
I20260620 02:52:33.403293 4980 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41605
I20260620 02:52:33.403651 4980 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:33.404361 4980 heartbeater.cc:511] Master 127.0.97.62:41605 requested a full tablet report, sending...
I20260620 02:52:33.406019 4526 ts_manager.cc:194] Registered new tserver with Master: 97aae3a894dd487bba5beddf22e9e261 (127.0.97.3:39669)
I20260620 02:52:33.407047 4526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48793
I20260620 02:52:33.414338 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:52:33.432336 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:33.419356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38358 (local address 127.0.97.62:41605)
0620 02:52:33.419522 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:33.419527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:33.420017 (+ 490us) server_negotiation.cc:407] Connection header received
0620 02:52:33.420213 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:33.420218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:33.420280 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:33.420370 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:33.421953 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:33.422790 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:33.424097 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:33.424549 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:33.424664 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:33.427098 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:33.427103 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:33.427107 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:33.427141 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:33.429893 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:33.430634 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:33.430638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:33.430641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:33.430759 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:33.431095 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:33.431098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:33.431101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:33.431413 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:33.431595 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:33.431914 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 02:52:33.432090 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 02:52:33.436894 4526 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:52:33.443610 4526 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:52:33.492060 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:33.456731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44357 (local address 127.0.0.1:51988)
0620 02:52:33.482568 (+ 25837us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:33.482594 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:33.482759 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:33.484304 (+ 1545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:33.484311 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:33.484339 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:33.484725 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:33.484736 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:33.486404 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:33.486410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:33.489218 (+ 2808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:33.489233 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:33.490962 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:33.490968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:33.491160 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:33.491238 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 02:52:33.491380 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 02:52:33.491607 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25615,"mutex_wait_us":589,"thread_start_us":25537,"threads_started":1}
I20260620 02:52:33.493009 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:33.457136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51988 (local address 127.0.97.2:44357)
0620 02:52:33.483567 (+ 26431us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:33.483580 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:33.483616 (+ 36us) server_negotiation.cc:407] Connection header received
0620 02:52:33.483738 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:33.483750 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:33.484050 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:33.484289 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:33.484909 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:33.486229 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:33.489432 (+ 3203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:33.490760 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:33.490887 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:33.491097 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:33.492465 (+ 1368us) server_negotiation.cc:299] Negotiation successful
0620 02:52:33.492668 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26222,"thread_start_us":26329,"threads_started":1}
I20260620 02:52:33.493583 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:33.456696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34766 (local address 127.0.97.3:39669)
0620 02:52:33.483554 (+ 26858us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:33.483564 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:33.483600 (+ 36us) server_negotiation.cc:407] Connection header received
0620 02:52:33.484608 (+ 1008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:33.484618 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:33.484859 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:33.485097 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:33.486010 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:33.487168 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:33.489548 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:33.491143 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:33.491274 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:33.491517 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:33.492160 (+ 643us) server_negotiation.cc:299] Negotiation successful
0620 02:52:33.493251 (+ 1091us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26671,"thread_start_us":26507,"threads_started":1}
I20260620 02:52:33.493115 4995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:33.456446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39669 (local address 127.0.0.1:34766)
0620 02:52:33.479697 (+ 23251us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:33.479776 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:33.480072 (+ 296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:33.485108 (+ 5036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:33.485115 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:33.485145 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:33.485778 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:33.485791 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:33.487312 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:33.487319 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:33.489388 (+ 2069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:33.489402 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:33.491366 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:33.491372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:33.491552 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:33.491588 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 02:52:33.491892 (+ 304us) client_negotiation.cc:240] Negotiation successful
0620 02:52:33.492049 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22830,"mutex_wait_us":55,"thread_start_us":25891,"threads_started":1}
I20260620 02:52:33.493731 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:33.456739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37300 (local address 127.0.97.1:46473)
0620 02:52:33.484137 (+ 27398us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:33.484148 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:33.484176 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:52:33.484269 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:33.484279 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:33.484517 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:33.484722 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:33.486049 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:33.487851 (+ 1802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:33.490658 (+ 2807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:33.492070 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:33.492190 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:33.492401 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:33.493206 (+ 805us) server_negotiation.cc:299] Negotiation successful
0620 02:52:33.493372 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27212,"thread_start_us":27038,"threads_started":1}
I20260620 02:52:33.494133 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:33.456445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46473 (local address 127.0.0.1:37300)
0620 02:52:33.482713 (+ 26268us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:33.482739 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:33.482895 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:33.484975 (+ 2080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:33.484980 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:33.485022 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:33.485779 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:33.485791 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:33.488053 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:33.488059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:33.490481 (+ 2422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:33.490504 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:33.492542 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:33.492548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:33.492729 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:33.492763 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 02:52:33.492850 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 02:52:33.492979 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25821,"mutex_wait_us":408,"thread_start_us":23604,"threads_started":1}
I20260620 02:52:33.497344 4782 tablet_service.cc:1467] Processing CreateTablet for tablet b5212bbbf59d44828b8e7c962d2f7aea (DEFAULT_TABLE table=test-table [id=0cdb80de82f0400aa891a5318d61dd35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:33.497345 4649 tablet_service.cc:1467] Processing CreateTablet for tablet b5212bbbf59d44828b8e7c962d2f7aea (DEFAULT_TABLE table=test-table [id=0cdb80de82f0400aa891a5318d61dd35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:33.497351 4915 tablet_service.cc:1467] Processing CreateTablet for tablet b5212bbbf59d44828b8e7c962d2f7aea (DEFAULT_TABLE table=test-table [id=0cdb80de82f0400aa891a5318d61dd35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:33.498725 4915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5212bbbf59d44828b8e7c962d2f7aea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:33.498749 4782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5212bbbf59d44828b8e7c962d2f7aea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:33.498749 4649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5212bbbf59d44828b8e7c962d2f7aea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:33.511708 4999 tablet_bootstrap.cc:492] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261: Bootstrap starting.
I20260620 02:52:33.514025 5001 tablet_bootstrap.cc:492] T b5212bbbf59d44828b8e7c962d2f7aea P cae0a74e639e4bc988864f8f8e5951f1: Bootstrap starting.
I20260620 02:52:33.514431 5000 tablet_bootstrap.cc:492] T b5212bbbf59d44828b8e7c962d2f7aea P 802b696d7e784ff680dca6822c264168: Bootstrap starting.
I20260620 02:52:33.515746 4999 tablet_bootstrap.cc:654] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:33.517271 4999 log.cc:824] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:33.518440 5000 tablet_bootstrap.cc:654] T b5212bbbf59d44828b8e7c962d2f7aea P 802b696d7e784ff680dca6822c264168: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:33.519773 5000 log.cc:824] T b5212bbbf59d44828b8e7c962d2f7aea P 802b696d7e784ff680dca6822c264168: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:33.521782 5001 tablet_bootstrap.cc:654] T b5212bbbf59d44828b8e7c962d2f7aea P cae0a74e639e4bc988864f8f8e5951f1: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:33.523108 5001 log.cc:824] T b5212bbbf59d44828b8e7c962d2f7aea P cae0a74e639e4bc988864f8f8e5951f1: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:33.523201 5000 tablet_bootstrap.cc:492] T b5212bbbf59d44828b8e7c962d2f7aea P 802b696d7e784ff680dca6822c264168: No bootstrap required, opened a new log
I20260620 02:52:33.523471 5000 ts_tablet_manager.cc:1397] T b5212bbbf59d44828b8e7c962d2f7aea P 802b696d7e784ff680dca6822c264168: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 02:52:33.523658 4999 tablet_bootstrap.cc:492] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261: No bootstrap required, opened a new log
I20260620 02:52:33.523914 4999 ts_tablet_manager.cc:1397] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.005s
I20260620 02:52:33.531648 4999 raft_consensus.cc:348] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97aae3a894dd487bba5beddf22e9e261" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39669 } } peers { permanent_uuid: "802b696d7e784ff680dca6822c264168" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46473 } } peers { permanent_uuid: "cae0a74e639e4bc988864f8f8e5951f1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44357 } }
I20260620 02:52:33.532045 4999 raft_consensus.cc:374] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:33.532210 4999 raft_consensus.cc:729] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97aae3a894dd487bba5beddf22e9e261, State: Initialized, Role: FOLLOWER
I20260620 02:52:33.532788 5001 tablet_bootstrap.cc:492] T b5212bbbf59d44828b8e7c962d2f7aea P cae0a74e639e4bc988864f8f8e5951f1: No bootstrap required, opened a new log
I20260620 02:52:33.532835 4999 consensus_queue.cc:260] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97aae3a894dd487bba5beddf22e9e261" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39669 } } peers { permanent_uuid: "802b696d7e784ff680dca6822c264168" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46473 } } peers { permanent_uuid: "cae0a74e639e4bc988864f8f8e5951f1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44357 } }
I20260620 02:52:33.533089 5001 ts_tablet_manager.cc:1397] T b5212bbbf59d44828b8e7c962d2f7aea P cae0a74e639e4bc988864f8f8e5951f1: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.005s
I20260620 02:52:33.533931 4980 heartbeater.cc:503] Master 127.0.97.62:41605 was elected leader, sending a full tablet report...
I20260620 02:52:33.535281 4999 ts_tablet_manager.cc:1428] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260620 02:52:33.536083 5000 raft_consensus.cc:348] T b5212bbbf59d44828b8e7c962d2f7aea P 802b696d7e784ff680dca6822c264168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97aae3a894dd487bba5beddf22e9e261" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39669 } } peers { permanent_uuid: "802b696d7e784ff680dca6822c264168" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46473 } } peers { permanent_uuid: "cae0a74e639e4bc988864f8f8e5951f1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44357 } }
I20260620 02:52:33.536502 5000 raft_consensus.cc:374] T b5212bbbf59d44828b8e7c962d2f7aea P 802b696d7e784ff680dca6822c264168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:33.536609 5000 raft_consensus.cc:729] T b5212bbbf59d44828b8e7c962d2f7aea P 802b696d7e784ff680dca6822c264168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 802b696d7e784ff680dca6822c264168, State: Initialized, Role: FOLLOWER
I20260620 02:52:33.537207 5000 consensus_queue.cc:260] T b5212bbbf59d44828b8e7c962d2f7aea P 802b696d7e784ff680dca6822c264168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97aae3a894dd487bba5beddf22e9e261" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39669 } } peers { permanent_uuid: "802b696d7e784ff680dca6822c264168" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46473 } } peers { permanent_uuid: "cae0a74e639e4bc988864f8f8e5951f1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44357 } }
I20260620 02:52:33.539500 5000 ts_tablet_manager.cc:1428] T b5212bbbf59d44828b8e7c962d2f7aea P 802b696d7e784ff680dca6822c264168: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260620 02:52:33.542718 5001 raft_consensus.cc:348] T b5212bbbf59d44828b8e7c962d2f7aea P cae0a74e639e4bc988864f8f8e5951f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97aae3a894dd487bba5beddf22e9e261" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39669 } } peers { permanent_uuid: "802b696d7e784ff680dca6822c264168" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46473 } } peers { permanent_uuid: "cae0a74e639e4bc988864f8f8e5951f1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44357 } }
I20260620 02:52:33.543087 5001 raft_consensus.cc:374] T b5212bbbf59d44828b8e7c962d2f7aea P cae0a74e639e4bc988864f8f8e5951f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:33.543211 5001 raft_consensus.cc:729] T b5212bbbf59d44828b8e7c962d2f7aea P cae0a74e639e4bc988864f8f8e5951f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae0a74e639e4bc988864f8f8e5951f1, State: Initialized, Role: FOLLOWER
I20260620 02:52:33.543787 5001 consensus_queue.cc:260] T b5212bbbf59d44828b8e7c962d2f7aea P cae0a74e639e4bc988864f8f8e5951f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97aae3a894dd487bba5beddf22e9e261" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39669 } } peers { permanent_uuid: "802b696d7e784ff680dca6822c264168" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46473 } } peers { permanent_uuid: "cae0a74e639e4bc988864f8f8e5951f1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44357 } }
I20260620 02:52:33.544760 4847 heartbeater.cc:503] Master 127.0.97.62:41605 was elected leader, sending a full tablet report...
I20260620 02:52:33.545202 5001 ts_tablet_manager.cc:1428] T b5212bbbf59d44828b8e7c962d2f7aea P cae0a74e639e4bc988864f8f8e5951f1: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
W20260620 02:52:33.582847 4715 tablet.cc:2369] T b5212bbbf59d44828b8e7c962d2f7aea P 802b696d7e784ff680dca6822c264168: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:52:33.609256 5005 raft_consensus.cc:482] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:52:33.609575 5005 raft_consensus.cc:504] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97aae3a894dd487bba5beddf22e9e261" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39669 } } peers { permanent_uuid: "802b696d7e784ff680dca6822c264168" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46473 } } peers { permanent_uuid: "cae0a74e639e4bc988864f8f8e5951f1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44357 } }
I20260620 02:52:33.611404 5005 leader_election.cc:288] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 802b696d7e784ff680dca6822c264168 (127.0.97.1:46473), cae0a74e639e4bc988864f8f8e5951f1 (127.0.97.2:44357)
W20260620 02:52:33.611609 4848 tablet.cc:2369] T b5212bbbf59d44828b8e7c962d2f7aea P cae0a74e639e4bc988864f8f8e5951f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:52:33.618588 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:33.611701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46473 (local address 127.0.97.3:54629)
0620 02:52:33.611947 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:33.611969 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:33.612361 (+ 392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:33.613389 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:33.613396 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:33.613423 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:33.613631 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:33.613640 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:33.614807 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:33.614812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:33.616427 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:33.616439 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:33.617798 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:33.617805 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:33.618014 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:33.618049 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 02:52:33.618145 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 02:52:33.618277 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":61}
I20260620 02:52:33.618686 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:33.612906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54629 (local address 127.0.97.1:46473)
0620 02:52:33.613127 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:33.613131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:33.613148 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:52:33.613213 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:33.613217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:33.613274 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:33.613370 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:33.613774 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:33.614687 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:33.616578 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:33.617635 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:33.617738 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:33.617807 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:33.618216 (+ 409us) server_negotiation.cc:299] Negotiation successful
0620 02:52:33.618359 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 02:52:33.619103 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:33.611935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43603 (local address 127.0.97.2:44357)
0620 02:52:33.612236 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:33.612245 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:33.613000 (+ 755us) server_negotiation.cc:407] Connection header received
0620 02:52:33.613262 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:33.613266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:33.613341 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:33.613433 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:33.614173 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:33.615043 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:33.617112 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:33.618037 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:33.618181 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:33.618269 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:33.618698 (+ 429us) server_negotiation.cc:299] Negotiation successful
0620 02:52:33.618850 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260620 02:52:33.619320 5008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:33.611715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44357 (local address 127.0.97.3:43603)
0620 02:52:33.612909 (+ 1194us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:33.612928 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:33.613113 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:33.613479 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:33.613484 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:33.613565 (+ 81us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:33.613971 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:33.613983 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:33.615185 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:33.615188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:33.616957 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:33.616973 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:33.618285 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:33.618292 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:33.618512 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:33.618551 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 02:52:33.618665 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 02:52:33.618806 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1066,"mutex_wait_us":392,"spinlock_wait_cycles":22912,"thread_start_us":490,"threads_started":1}
I20260620 02:52:33.619863 4669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5212bbbf59d44828b8e7c962d2f7aea" candidate_uuid: "97aae3a894dd487bba5beddf22e9e261" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "802b696d7e784ff680dca6822c264168" is_pre_election: true
I20260620 02:52:33.620344 4669 raft_consensus.cc:2454] T b5212bbbf59d44828b8e7c962d2f7aea P 802b696d7e784ff680dca6822c264168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97aae3a894dd487bba5beddf22e9e261 in term 0.
I20260620 02:52:33.620265 4802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5212bbbf59d44828b8e7c962d2f7aea" candidate_uuid: "97aae3a894dd487bba5beddf22e9e261" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae0a74e639e4bc988864f8f8e5951f1" is_pre_election: true
I20260620 02:52:33.620780 4802 raft_consensus.cc:2454] T b5212bbbf59d44828b8e7c962d2f7aea P cae0a74e639e4bc988864f8f8e5951f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97aae3a894dd487bba5beddf22e9e261 in term 0.
I20260620 02:52:33.621066 4868 leader_election.cc:302] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 802b696d7e784ff680dca6822c264168, 97aae3a894dd487bba5beddf22e9e261; no voters:
I20260620 02:52:33.621425 5005 raft_consensus.cc:2781] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:52:33.621589 5005 raft_consensus.cc:482] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:52:33.621692 5005 raft_consensus.cc:3037] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:33.623831 5005 raft_consensus.cc:504] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97aae3a894dd487bba5beddf22e9e261" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39669 } } peers { permanent_uuid: "802b696d7e784ff680dca6822c264168" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46473 } } peers { permanent_uuid: "cae0a74e639e4bc988864f8f8e5951f1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44357 } }
I20260620 02:52:33.624424 5005 leader_election.cc:288] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261 [CANDIDATE]: Term 1 election: Requested vote from peers 802b696d7e784ff680dca6822c264168 (127.0.97.1:46473), cae0a74e639e4bc988864f8f8e5951f1 (127.0.97.2:44357)
I20260620 02:52:33.624821 4669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5212bbbf59d44828b8e7c962d2f7aea" candidate_uuid: "97aae3a894dd487bba5beddf22e9e261" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "802b696d7e784ff680dca6822c264168"
I20260620 02:52:33.625032 4669 raft_consensus.cc:3037] T b5212bbbf59d44828b8e7c962d2f7aea P 802b696d7e784ff680dca6822c264168 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:33.625077 4802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5212bbbf59d44828b8e7c962d2f7aea" candidate_uuid: "97aae3a894dd487bba5beddf22e9e261" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae0a74e639e4bc988864f8f8e5951f1"
I20260620 02:52:33.625322 4802 raft_consensus.cc:3037] T b5212bbbf59d44828b8e7c962d2f7aea P cae0a74e639e4bc988864f8f8e5951f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:33.627264 4669 raft_consensus.cc:2454] T b5212bbbf59d44828b8e7c962d2f7aea P 802b696d7e784ff680dca6822c264168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97aae3a894dd487bba5beddf22e9e261 in term 1.
I20260620 02:52:33.627663 4868 leader_election.cc:302] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 802b696d7e784ff680dca6822c264168, 97aae3a894dd487bba5beddf22e9e261; no voters:
I20260620 02:52:33.627743 4802 raft_consensus.cc:2454] T b5212bbbf59d44828b8e7c962d2f7aea P cae0a74e639e4bc988864f8f8e5951f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97aae3a894dd487bba5beddf22e9e261 in term 1.
I20260620 02:52:33.627990 5005 raft_consensus.cc:2781] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:33.628532 5005 raft_consensus.cc:686] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261 [term 1 LEADER]: Becoming Leader. State: Replica: 97aae3a894dd487bba5beddf22e9e261, State: Running, Role: LEADER
I20260620 02:52:33.629024 5005 consensus_queue.cc:237] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97aae3a894dd487bba5beddf22e9e261" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39669 } } peers { permanent_uuid: "802b696d7e784ff680dca6822c264168" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46473 } } peers { permanent_uuid: "cae0a74e639e4bc988864f8f8e5951f1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44357 } }
W20260620 02:52:33.631029 4981 tablet.cc:2369] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:52:33.633652 4526 catalog_manager.cc:5495] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97aae3a894dd487bba5beddf22e9e261 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "97aae3a894dd487bba5beddf22e9e261" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97aae3a894dd487bba5beddf22e9e261" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "802b696d7e784ff680dca6822c264168" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cae0a74e639e4bc988864f8f8e5951f1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44357 } health_report { overall_health: UNKNOWN } } }
I20260620 02:52:33.684226 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:33.678379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34768 (local address 127.0.97.3:39669)
0620 02:52:33.678617 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:33.678621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:33.678638 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:52:33.678840 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:33.678844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:33.678951 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:33.679078 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:52:33.679489 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:33.680365 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:33.681869 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:33.682316 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:33.682534 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:33.683069 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:52:33.683600 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:52:33.683791 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:33.683843 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 02:52:33.683927 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 02:52:33.693019 4802 raft_consensus.cc:1261] T b5212bbbf59d44828b8e7c962d2f7aea P cae0a74e639e4bc988864f8f8e5951f1 [term 1 FOLLOWER]: Refusing update from remote peer 97aae3a894dd487bba5beddf22e9e261: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:33.693020 4669 raft_consensus.cc:1261] T b5212bbbf59d44828b8e7c962d2f7aea P 802b696d7e784ff680dca6822c264168 [term 1 FOLLOWER]: Refusing update from remote peer 97aae3a894dd487bba5beddf22e9e261: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:33.694017 5005 consensus_queue.cc:1035] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae0a74e639e4bc988864f8f8e5951f1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:52:33.694316 5009 consensus_queue.cc:1035] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261 [LEADER]: Connected to new peer: Peer: permanent_uuid: "802b696d7e784ff680dca6822c264168" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:52:33.707332 5012 mvcc.cc:205] Tried to move back new op lower bound from 7298760514315689984 to 7298760514068381696. Current Snapshot: MvccSnapshot[applied={T|T < 7298760514315689984}]
I20260620 02:52:33.712479 5014 mvcc.cc:205] Tried to move back new op lower bound from 7298760514315689984 to 7298760514068381696. Current Snapshot: MvccSnapshot[applied={T|T < 7298760514315689984}]
I20260620 02:52:33.712890 5013 mvcc.cc:205] Tried to move back new op lower bound from 7298760514315689984 to 7298760514068381696. Current Snapshot: MvccSnapshot[applied={T|T < 7298760514315689984}]
I20260620 02:52:33.723349 4526 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:52:33.723716 4526 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:52:33.727993 4526 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e0c374527c340c78eec39261b2cff36: Sending DeleteTablet for 3 replicas of tablet b5212bbbf59d44828b8e7c962d2f7aea
I20260620 02:52:33.729061 4915 tablet_service.cc:1514] Processing DeleteTablet for tablet b5212bbbf59d44828b8e7c962d2f7aea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34766
I20260620 02:52:33.729202 4782 tablet_service.cc:1514] Processing DeleteTablet for tablet b5212bbbf59d44828b8e7c962d2f7aea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51988
I20260620 02:52:33.730551 4649 tablet_service.cc:1514] Processing DeleteTablet for tablet b5212bbbf59d44828b8e7c962d2f7aea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37300
I20260620 02:52:33.735239 5023 tablet_replica.cc:330] T b5212bbbf59d44828b8e7c962d2f7aea P cae0a74e639e4bc988864f8f8e5951f1: stopping tablet replica
I20260620 02:52:33.735651 5023 raft_consensus.cc:2229] T b5212bbbf59d44828b8e7c962d2f7aea P cae0a74e639e4bc988864f8f8e5951f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:52:33.736095 5023 raft_consensus.cc:2258] T b5212bbbf59d44828b8e7c962d2f7aea P cae0a74e639e4bc988864f8f8e5951f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:33.736754 5022 tablet_replica.cc:330] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261: stopping tablet replica
I20260620 02:52:33.736953 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 4587
I20260620 02:52:33.737097 5022 raft_consensus.cc:2229] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:52:33.737715 5022 raft_consensus.cc:2258] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:33.745900 5024 tablet_replica.cc:330] T b5212bbbf59d44828b8e7c962d2f7aea P 802b696d7e784ff680dca6822c264168: stopping tablet replica
I20260620 02:52:33.746403 5024 raft_consensus.cc:2229] T b5212bbbf59d44828b8e7c962d2f7aea P 802b696d7e784ff680dca6822c264168 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:52:33.746978 5024 raft_consensus.cc:2258] T b5212bbbf59d44828b8e7c962d2f7aea P 802b696d7e784ff680dca6822c264168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:33.747004 5022 ts_tablet_manager.cc:1905] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:33.749159 5024 ts_tablet_manager.cc:1905] T b5212bbbf59d44828b8e7c962d2f7aea P 802b696d7e784ff680dca6822c264168: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:33.743878 5023 ts_tablet_manager.cc:1905] T b5212bbbf59d44828b8e7c962d2f7aea P cae0a74e639e4bc988864f8f8e5951f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:33.756091 5022 ts_tablet_manager.cc:1918] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:52:33.756358 5022 log.cc:1196] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-2/wal/wals/b5212bbbf59d44828b8e7c962d2f7aea
I20260620 02:52:33.756568 5024 ts_tablet_manager.cc:1918] T b5212bbbf59d44828b8e7c962d2f7aea P 802b696d7e784ff680dca6822c264168: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:52:33.756783 5024 log.cc:1196] T b5212bbbf59d44828b8e7c962d2f7aea P 802b696d7e784ff680dca6822c264168: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-0/wal/wals/b5212bbbf59d44828b8e7c962d2f7aea
I20260620 02:52:33.757097 5022 ts_tablet_manager.cc:1939] T b5212bbbf59d44828b8e7c962d2f7aea P 97aae3a894dd487bba5beddf22e9e261: Deleting consensus metadata
I20260620 02:52:33.757344 5024 ts_tablet_manager.cc:1939] T b5212bbbf59d44828b8e7c962d2f7aea P 802b696d7e784ff680dca6822c264168: Deleting consensus metadata
I20260620 02:52:33.759238 4510 catalog_manager.cc:4841] TS 802b696d7e784ff680dca6822c264168 (127.0.97.1:46473): tablet b5212bbbf59d44828b8e7c962d2f7aea (table test-table [id=0cdb80de82f0400aa891a5318d61dd35]) successfully deleted
I20260620 02:52:33.759500 4511 catalog_manager.cc:4841] TS 97aae3a894dd487bba5beddf22e9e261 (127.0.97.3:39669): tablet b5212bbbf59d44828b8e7c962d2f7aea (table test-table [id=0cdb80de82f0400aa891a5318d61dd35]) successfully deleted
I20260620 02:52:33.762099 5023 ts_tablet_manager.cc:1918] T b5212bbbf59d44828b8e7c962d2f7aea P cae0a74e639e4bc988864f8f8e5951f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:52:33.762282 5023 log.cc:1196] T b5212bbbf59d44828b8e7c962d2f7aea P cae0a74e639e4bc988864f8f8e5951f1: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781923914418765-388-0/minicluster-data/ts-1/wal/wals/b5212bbbf59d44828b8e7c962d2f7aea
I20260620 02:52:33.762892 5023 ts_tablet_manager.cc:1939] T b5212bbbf59d44828b8e7c962d2f7aea P cae0a74e639e4bc988864f8f8e5951f1: Deleting consensus metadata
I20260620 02:52:33.764703 4513 catalog_manager.cc:4841] TS cae0a74e639e4bc988864f8f8e5951f1 (127.0.97.2:44357): tablet b5212bbbf59d44828b8e7c962d2f7aea (table test-table [id=0cdb80de82f0400aa891a5318d61dd35]) successfully deleted
Jun 20 02:52:33 dist-test-slave-8p8c krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:52:33.776172 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:33.738114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37304 (local address 127.0.97.1:46473)
0620 02:52:33.738294 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:33.738301 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:33.759690 (+ 21389us) server_negotiation.cc:407] Connection header received
0620 02:52:33.759981 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:33.759987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:33.760084 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:33.760192 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:33.761845 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:33.762987 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:33.764790 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:33.765409 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:33.765553 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:33.769738 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:33.769754 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:33.769776 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:33.769827 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:33.773289 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:33.773932 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:33.773941 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:33.773945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:33.774107 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:33.774506 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:33.774512 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:33.774515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:33.774942 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:33.775264 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:33.775564 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 02:52:33.775774 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 02:52:33.776932 4709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:33.866976 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 4587
I20260620 02:52:33.884682 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 4720
Jun 20 02:52:33 dist-test-slave-8p8c krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:52:33.901117 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:33.885700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51998 (local address 127.0.97.2:44357)
0620 02:52:33.885872 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:33.885878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:33.885896 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:52:33.885955 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:33.885959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:33.886035 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:33.886156 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:33.887376 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:33.888153 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:33.889488 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:33.890062 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:33.890193 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:33.894786 (+ 4593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:33.894800 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:33.894815 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:33.894853 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:33.897925 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:33.898842 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:33.898853 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:33.898857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:33.899008 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:33.899414 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:33.899420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:33.899423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:33.899830 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:33.900152 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:33.900509 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 02:52:33.900762 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 02:52:33.901777 4842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:33.995424 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 4720
I20260620 02:52:34.012552 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 4853
Jun 20 02:52:34 dist-test-slave-8p8c krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:52:34.030769 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:34.013679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34780 (local address 127.0.97.3:39669)
0620 02:52:34.013896 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:34.013901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:34.013923 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:52:34.014113 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:34.014117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:34.014182 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:34.014305 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:34.015590 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:34.016584 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:34.018044 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:34.018629 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:34.018789 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:34.023851 (+ 5062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:34.023869 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:34.023885 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:34.023936 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:34.027425 (+ 3489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:34.028168 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:34.028299 (+ 131us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:34.028302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:34.028495 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:34.029005 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:34.029009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:34.029011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:34.029444 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:34.029809 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:34.030112 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 02:52:34.030348 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 02:52:34.031510 4975 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:34.153077 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 4853
I20260620 02:52:34.172688 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 4495
I20260620 02:52:34.173902 4556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:34.285426 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 4495
2026-06-20T02:52:34Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4360 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T02:52:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:52:34Z Disabled control of system clock
I20260620 02:52:34.356062 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35501
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 02:52:34.683362 5033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:34.683660 5033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:34.683830 5033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:34.691110 5033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:52:34.691228 5033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:34.691265 5033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:34.691380 5033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:52:34.691466 5033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:52:34.700719 5033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35501
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:34.702059 5033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:34.703913 5033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:34.716007 5038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:34.716193 5039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:34.716629 5041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:34.717475 5033 server_base.cc:1034] running on GCE node
I20260620 02:52:34.718328 5033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:34.719898 5033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:34.721168 5033 hybrid_clock.cc:648] HybridClock initialized: now 1781923954721147 us; error 62 us; skew 500 ppm
I20260620 02:52:34.721582 5033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:52:34.724027 5033 webserver.cc:460] Webserver started at http://127.0.97.62:42349/ using document root <none> and password file <none>
I20260620 02:52:34.724759 5033 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:34.724893 5033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:34.725229 5033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:34.728421 5033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "f48e06309e6843ad8d88979564a35bb9"
format_stamp: "Formatted at 2026-06-20 02:52:34 on dist-test-slave-8p8c"
server_key: "0625d6baa9343f55c071fabe05723131"
server_key_iv: "6b2432b648e55f1695f29e873c2f68e7"
server_key_version: "encryptionkey@0"
I20260620 02:52:34.729463 5033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "f48e06309e6843ad8d88979564a35bb9"
format_stamp: "Formatted at 2026-06-20 02:52:34 on dist-test-slave-8p8c"
server_key: "0625d6baa9343f55c071fabe05723131"
server_key_iv: "6b2432b648e55f1695f29e873c2f68e7"
server_key_version: "encryptionkey@0"
I20260620 02:52:34.735225 5033 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 02:52:34.739078 5047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:34.740628 5033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:52:34.740845 5033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "f48e06309e6843ad8d88979564a35bb9"
format_stamp: "Formatted at 2026-06-20 02:52:34 on dist-test-slave-8p8c"
server_key: "0625d6baa9343f55c071fabe05723131"
server_key_iv: "6b2432b648e55f1695f29e873c2f68e7"
server_key_version: "encryptionkey@0"
I20260620 02:52:34.741099 5033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:34.747583 5033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:34.748505 5033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:34.748893 5033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:34.769902 5033 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35501
I20260620 02:52:34.769922 5098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35501 every 8 connection(s)
I20260620 02:52:34.771384 5033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:52:34.775193 5099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:34.776028 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 5033
I20260620 02:52:34.776230 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:52:34.777045 388 external_mini_cluster.cc:1428] Setting key 2c0ffc90831e157fea5bd0942f581b1b
I20260620 02:52:34.785614 5102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:34.779351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41686 (local address 127.0.97.62:35501)
0620 02:52:34.780380 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:34.780391 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:34.780434 (+ 43us) server_negotiation.cc:407] Connection header received
0620 02:52:34.782050 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:34.782074 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:34.782577 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:34.782988 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:34.784026 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:34.784032 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:34.784044 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:52:34.784060 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:34.784115 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:52:34.784130 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:34.784374 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:34.784505 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 02:52:34.784745 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":357,"threads_started":1}
I20260620 02:52:34.788730 5099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f48e06309e6843ad8d88979564a35bb9: Bootstrap starting.
I20260620 02:52:34.791707 5099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f48e06309e6843ad8d88979564a35bb9: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:34.792847 5099 log.cc:824] T 00000000000000000000000000000000 P f48e06309e6843ad8d88979564a35bb9: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:34.795495 5099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f48e06309e6843ad8d88979564a35bb9: No bootstrap required, opened a new log
I20260620 02:52:34.802800 5099 raft_consensus.cc:348] T 00000000000000000000000000000000 P f48e06309e6843ad8d88979564a35bb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48e06309e6843ad8d88979564a35bb9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35501 } }
I20260620 02:52:34.803169 5099 raft_consensus.cc:374] T 00000000000000000000000000000000 P f48e06309e6843ad8d88979564a35bb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:34.803272 5099 raft_consensus.cc:729] T 00000000000000000000000000000000 P f48e06309e6843ad8d88979564a35bb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f48e06309e6843ad8d88979564a35bb9, State: Initialized, Role: FOLLOWER
I20260620 02:52:34.803886 5099 consensus_queue.cc:260] T 00000000000000000000000000000000 P f48e06309e6843ad8d88979564a35bb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48e06309e6843ad8d88979564a35bb9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35501 } }
I20260620 02:52:34.804301 5099 raft_consensus.cc:388] T 00000000000000000000000000000000 P f48e06309e6843ad8d88979564a35bb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:52:34.804450 5099 raft_consensus.cc:482] T 00000000000000000000000000000000 P f48e06309e6843ad8d88979564a35bb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:52:34.804622 5099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f48e06309e6843ad8d88979564a35bb9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:34.806789 5099 raft_consensus.cc:504] T 00000000000000000000000000000000 P f48e06309e6843ad8d88979564a35bb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48e06309e6843ad8d88979564a35bb9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35501 } }
I20260620 02:52:34.807282 5099 leader_election.cc:302] T 00000000000000000000000000000000 P f48e06309e6843ad8d88979564a35bb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f48e06309e6843ad8d88979564a35bb9; no voters:
I20260620 02:52:34.807857 5099 leader_election.cc:288] T 00000000000000000000000000000000 P f48e06309e6843ad8d88979564a35bb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:52:34.808034 5104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f48e06309e6843ad8d88979564a35bb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:34.809091 5104 raft_consensus.cc:686] T 00000000000000000000000000000000 P f48e06309e6843ad8d88979564a35bb9 [term 1 LEADER]: Becoming Leader. State: Replica: f48e06309e6843ad8d88979564a35bb9, State: Running, Role: LEADER
I20260620 02:52:34.809607 5104 consensus_queue.cc:237] T 00000000000000000000000000000000 P f48e06309e6843ad8d88979564a35bb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48e06309e6843ad8d88979564a35bb9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35501 } }
I20260620 02:52:34.810220 5099 sys_catalog.cc:564] T 00000000000000000000000000000000 P f48e06309e6843ad8d88979564a35bb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:52:34.812423 5105 sys_catalog.cc:455] T 00000000000000000000000000000000 P f48e06309e6843ad8d88979564a35bb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f48e06309e6843ad8d88979564a35bb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48e06309e6843ad8d88979564a35bb9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35501 } } }
I20260620 02:52:34.812556 5106 sys_catalog.cc:455] T 00000000000000000000000000000000 P f48e06309e6843ad8d88979564a35bb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f48e06309e6843ad8d88979564a35bb9. Latest consensus state: current_term: 1 leader_uuid: "f48e06309e6843ad8d88979564a35bb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48e06309e6843ad8d88979564a35bb9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35501 } } }
I20260620 02:52:34.812856 5106 sys_catalog.cc:458] T 00000000000000000000000000000000 P f48e06309e6843ad8d88979564a35bb9 [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:34.812853 5105 sys_catalog.cc:458] T 00000000000000000000000000000000 P f48e06309e6843ad8d88979564a35bb9 [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:34.813879 5112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:52:34.820338 5112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:52:34.831403 5112 catalog_manager.cc:1347] Generated new cluster ID: 1775087a32c148d893e2131fb6b0dc84
I20260620 02:52:34.831548 5112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:52:34.844498 5112 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:52:34.844997 5112 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:52:34.851033 5112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f48e06309e6843ad8d88979564a35bb9: Generated new TSK 0
I20260620 02:52:34.851833 5112 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:52:34.860092 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35501
--builtin_ntp_servers=127.0.97.20:34013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 02:52:35.194916 5123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:35.195226 5123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:35.195317 5123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:35.202561 5123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:35.202657 5123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:35.202764 5123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:52:35.211750 5123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:35.212981 5123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:35.214893 5123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:35.228287 5129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:35.228230 5128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:35.228310 5131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:35.229110 5123 server_base.cc:1034] running on GCE node
I20260620 02:52:35.229655 5123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:35.230615 5123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:35.231835 5123 hybrid_clock.cc:648] HybridClock initialized: now 1781923955231763 us; error 57 us; skew 500 ppm
I20260620 02:52:35.232136 5123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:52:35.234287 5123 webserver.cc:460] Webserver started at http://127.0.97.1:46025/ using document root <none> and password file <none>
I20260620 02:52:35.235066 5123 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:35.235153 5123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:35.235412 5123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:35.238147 5123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-0/data/instance:
uuid: "041d670e51c74fc9a970667ff4fa67d1"
format_stamp: "Formatted at 2026-06-20 02:52:35 on dist-test-slave-8p8c"
server_key: "9ca388914fae908bedfcc74fcbd020d5"
server_key_iv: "4a9b14cd2ec1aa95550c04c4ea6268c2"
server_key_version: "encryptionkey@0"
I20260620 02:52:35.238999 5123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-0/wal/instance:
uuid: "041d670e51c74fc9a970667ff4fa67d1"
format_stamp: "Formatted at 2026-06-20 02:52:35 on dist-test-slave-8p8c"
server_key: "9ca388914fae908bedfcc74fcbd020d5"
server_key_iv: "4a9b14cd2ec1aa95550c04c4ea6268c2"
server_key_version: "encryptionkey@0"
I20260620 02:52:35.243875 5123 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:52:35.247433 5137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:35.248781 5123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 02:52:35.248917 5123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-0/wal
uuid: "041d670e51c74fc9a970667ff4fa67d1"
format_stamp: "Formatted at 2026-06-20 02:52:35 on dist-test-slave-8p8c"
server_key: "9ca388914fae908bedfcc74fcbd020d5"
server_key_iv: "4a9b14cd2ec1aa95550c04c4ea6268c2"
server_key_version: "encryptionkey@0"
I20260620 02:52:35.249112 5123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:35.254770 5123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:35.255477 5123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:35.255690 5123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:35.256517 5123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:52:35.257819 5123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:35.257905 5123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:35.257975 5123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:35.258041 5123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:35.291175 5123 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34957
I20260620 02:52:35.291249 5249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34957 every 8 connection(s)
I20260620 02:52:35.292484 5123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:52:35.300508 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 5123
I20260620 02:52:35.300693 5252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:35.295024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59477 (local address 127.0.97.62:35501)
0620 02:52:35.295455 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:35.295459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:35.296734 (+ 1275us) server_negotiation.cc:407] Connection header received
0620 02:52:35.298461 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:35.298467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:35.298533 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:35.298661 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:35.299927 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:35.299933 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:35.299938 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:52:35.299964 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:35.300013 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:52:35.300026 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:35.300148 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:35.300344 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 02:52:35.300409 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":234,"threads_started":1}
I20260620 02:52:35.300719 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:52:35.301388 388 external_mini_cluster.cc:1428] Setting key b689a2bb6584baa1c7d6ed65e1fa0aff
I20260620 02:52:35.301657 5253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:35.295631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35501 (local address 127.0.97.1:59477)
0620 02:52:35.296570 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:35.296611 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:35.298167 (+ 1556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:35.298893 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:35.298903 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:35.298996 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:35.299631 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:52:35.299634 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:35.299669 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:52:35.299751 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:35.300120 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:35.300130 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:35.300137 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 02:52:35.300410 (+ 273us) client_negotiation.cc:240] Negotiation successful
0620 02:52:35.300689 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":282,"threads_started":1}
I20260620 02:52:35.303167 5250 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35501
I20260620 02:52:35.303484 5250 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:35.304052 5250 heartbeater.cc:511] Master 127.0.97.62:35501 requested a full tablet report, sending...
I20260620 02:52:35.304450 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35501
--builtin_ntp_servers=127.0.97.20:34013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 02:52:35.306092 5063 ts_manager.cc:194] Registered new tserver with Master: 041d670e51c74fc9a970667ff4fa67d1 (127.0.97.1:34957)
W20260620 02:52:35.636175 5254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:35.636490 5254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:35.636654 5254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:35.644014 5254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:35.644117 5254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:35.644289 5254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:52:35.654342 5254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:35.655727 5254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:35.657543 5254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:35.671041 5259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:35.671119 5262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:35.671144 5260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:35.672269 5254 server_base.cc:1034] running on GCE node
I20260620 02:52:35.672976 5254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:35.673913 5254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:35.675187 5254 hybrid_clock.cc:648] HybridClock initialized: now 1781923955675118 us; error 68 us; skew 500 ppm
I20260620 02:52:35.675480 5254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:52:35.677929 5254 webserver.cc:460] Webserver started at http://127.0.97.2:43167/ using document root <none> and password file <none>
I20260620 02:52:35.678788 5254 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:35.678885 5254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:35.679159 5254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:35.682325 5254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-1/data/instance:
uuid: "a17c3994f2944e9297649ca936d551bb"
format_stamp: "Formatted at 2026-06-20 02:52:35 on dist-test-slave-8p8c"
server_key: "ff99cf2c4498b2293a6ef26576ef9088"
server_key_iv: "8228e698b2c718ff4391bdf2f2985d48"
server_key_version: "encryptionkey@0"
I20260620 02:52:35.683457 5254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a17c3994f2944e9297649ca936d551bb"
format_stamp: "Formatted at 2026-06-20 02:52:35 on dist-test-slave-8p8c"
server_key: "ff99cf2c4498b2293a6ef26576ef9088"
server_key_iv: "8228e698b2c718ff4391bdf2f2985d48"
server_key_version: "encryptionkey@0"
I20260620 02:52:35.689625 5254 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 02:52:35.693661 5268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:35.695363 5254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:52:35.695577 5254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-1/wal
uuid: "a17c3994f2944e9297649ca936d551bb"
format_stamp: "Formatted at 2026-06-20 02:52:35 on dist-test-slave-8p8c"
server_key: "ff99cf2c4498b2293a6ef26576ef9088"
server_key_iv: "8228e698b2c718ff4391bdf2f2985d48"
server_key_version: "encryptionkey@0"
I20260620 02:52:35.695833 5254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:35.702054 5254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:35.702986 5254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:35.703305 5254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:35.704304 5254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:52:35.705793 5254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:35.705917 5254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:35.706049 5254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:35.706140 5254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:35.746001 5254 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45649
I20260620 02:52:35.746100 5380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45649 every 8 connection(s)
I20260620 02:52:35.747855 5254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:52:35.754632 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 5254
I20260620 02:52:35.754838 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:52:35.755487 388 external_mini_cluster.cc:1428] Setting key d5b3e5066eb298031044d84f5cc5baa2
I20260620 02:52:35.757773 5252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:35.750561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48607 (local address 127.0.97.62:35501)
0620 02:52:35.750774 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:35.750781 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:35.752245 (+ 1464us) server_negotiation.cc:407] Connection header received
0620 02:52:35.754886 (+ 2641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:35.754895 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:35.754988 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:35.755119 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:35.756853 (+ 1734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:35.756859 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:35.756864 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:52:35.756885 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:35.756977 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:52:35.756997 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:35.757138 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:35.757326 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 02:52:35.757387 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 02:52:35.758800 5383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:35.751146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35501 (local address 127.0.97.2:48607)
0620 02:52:35.752076 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:35.752112 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:35.754327 (+ 2215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:35.755408 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:35.755422 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:35.755587 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:35.756425 (+ 838us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:52:35.756429 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:35.756506 (+ 77us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:52:35.756631 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:35.757120 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:35.757128 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:35.757135 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 02:52:35.757375 (+ 240us) client_negotiation.cc:240] Negotiation successful
0620 02:52:35.757655 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":282,"threads_started":1}
I20260620 02:52:35.758949 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35501
--builtin_ntp_servers=127.0.97.20:34013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 02:52:35.760396 5381 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35501
I20260620 02:52:35.760673 5381 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:35.761227 5381 heartbeater.cc:511] Master 127.0.97.62:35501 requested a full tablet report, sending...
I20260620 02:52:35.762805 5063 ts_manager.cc:194] Registered new tserver with Master: a17c3994f2944e9297649ca936d551bb (127.0.97.2:45649)
W20260620 02:52:36.103199 5384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:36.103482 5384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:36.103633 5384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:36.110918 5384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:36.111028 5384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:36.111192 5384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:52:36.120512 5384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:36.121809 5384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:36.123641 5384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:36.137791 5390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:36.137784 5392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:36.137965 5389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:36.138844 5384 server_base.cc:1034] running on GCE node
I20260620 02:52:36.139587 5384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:36.140745 5384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:36.142040 5384 hybrid_clock.cc:648] HybridClock initialized: now 1781923956141954 us; error 70 us; skew 500 ppm
I20260620 02:52:36.142472 5384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:52:36.145087 5384 webserver.cc:460] Webserver started at http://127.0.97.3:38125/ using document root <none> and password file <none>
I20260620 02:52:36.145915 5384 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:36.146090 5384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:36.146512 5384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:36.149582 5384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-2/data/instance:
uuid: "b2197c40b54b49bc9f47cedb2991c7b5"
format_stamp: "Formatted at 2026-06-20 02:52:36 on dist-test-slave-8p8c"
server_key: "345415ab31acbb9255599e1ed248dcb8"
server_key_iv: "bb50629fa2c267a3c58e2feb1317efda"
server_key_version: "encryptionkey@0"
I20260620 02:52:36.150490 5384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b2197c40b54b49bc9f47cedb2991c7b5"
format_stamp: "Formatted at 2026-06-20 02:52:36 on dist-test-slave-8p8c"
server_key: "345415ab31acbb9255599e1ed248dcb8"
server_key_iv: "bb50629fa2c267a3c58e2feb1317efda"
server_key_version: "encryptionkey@0"
I20260620 02:52:36.156262 5384 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 02:52:36.159907 5398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:36.161444 5384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:52:36.161645 5384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-2/wal
uuid: "b2197c40b54b49bc9f47cedb2991c7b5"
format_stamp: "Formatted at 2026-06-20 02:52:36 on dist-test-slave-8p8c"
server_key: "345415ab31acbb9255599e1ed248dcb8"
server_key_iv: "bb50629fa2c267a3c58e2feb1317efda"
server_key_version: "encryptionkey@0"
I20260620 02:52:36.161895 5384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:36.167815 5384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:36.168666 5384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:36.168951 5384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:36.169898 5384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:52:36.171494 5384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:36.171619 5384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:36.171756 5384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:36.171842 5384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:36.206353 5384 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35045
I20260620 02:52:36.206480 5510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35045 every 8 connection(s)
I20260620 02:52:36.207751 5384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:52:36.210747 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 5384
I20260620 02:52:36.210909 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:52:36.211334 388 external_mini_cluster.cc:1428] Setting key 1e7e3f811b8691b87f73b434f862f692
I20260620 02:52:36.217144 5252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:36.210601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37625 (local address 127.0.97.62:35501)
0620 02:52:36.210825 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:36.210832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:36.212695 (+ 1863us) server_negotiation.cc:407] Connection header received
0620 02:52:36.215032 (+ 2337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:36.215038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:36.215106 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:36.215194 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:36.216479 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:36.216483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:36.216487 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:52:36.216503 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:36.216533 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:52:36.216544 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:36.216649 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:36.216850 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 02:52:36.216898 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:52:36.218024 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:36.211331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35501 (local address 127.0.97.3:37625)
0620 02:52:36.212540 (+ 1209us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:36.212590 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:36.214744 (+ 2154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:36.215410 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:36.215418 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:36.215515 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:36.216186 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:52:36.216189 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:36.216230 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:52:36.216315 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:36.216643 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:36.216650 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:36.216655 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 02:52:36.216864 (+ 209us) client_negotiation.cc:240] Negotiation successful
0620 02:52:36.217091 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":377,"threads_started":1}
I20260620 02:52:36.219736 5511 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35501
I20260620 02:52:36.220108 5511 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:36.221028 5511 heartbeater.cc:511] Master 127.0.97.62:35501 requested a full tablet report, sending...
I20260620 02:52:36.222836 5063 ts_manager.cc:194] Registered new tserver with Master: b2197c40b54b49bc9f47cedb2991c7b5 (127.0.97.3:35045)
I20260620 02:52:36.226696 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:52:36.234225 5252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:36.231492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41700 (local address 127.0.97.62:35501)
0620 02:52:36.231691 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:36.231738 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:36.232211 (+ 473us) server_negotiation.cc:407] Connection header received
0620 02:52:36.232423 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:36.232428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:36.232497 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:36.232609 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:36.233591 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:36.233600 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:36.233605 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:52:36.233623 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:36.233660 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:52:36.233680 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:36.233806 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:36.233885 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 02:52:36.233944 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 02:52:36.238472 5063 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:52:36.244910 5063 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:52:36.263576 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:36.258093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34957 (local address 127.0.0.1:59872)
0620 02:52:36.258913 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:36.258973 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:36.259124 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:36.260857 (+ 1733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:36.260867 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:36.261023 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:36.261959 (+ 936us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:52:36.261962 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:36.261998 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:52:36.262078 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:36.262659 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:36.262677 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:36.262684 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 02:52:36.262891 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 02:52:36.263000 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":365,"threads_started":1}
I20260620 02:52:36.264305 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:36.258245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59872 (local address 127.0.97.1:34957)
0620 02:52:36.258896 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:36.258907 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:36.259270 (+ 363us) server_negotiation.cc:407] Connection header received
0620 02:52:36.259351 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:36.259362 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:36.259630 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:36.259804 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:36.262279 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:36.262289 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:36.262304 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:52:36.262335 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:36.262438 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:52:36.262463 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:36.262603 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:36.263739 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0620 02:52:36.263856 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":804,"threads_started":1}
I20260620 02:52:36.265621 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:36.260076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35045 (local address 127.0.0.1:56190)
0620 02:52:36.260662 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:36.260692 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:36.260801 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:36.263664 (+ 2863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:36.263670 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:36.263695 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:36.264384 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:52:36.264388 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:36.264406 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:52:36.264467 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:36.264902 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:36.264907 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:36.264913 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 02:52:36.265052 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 02:52:36.265104 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":649,"threads_started":1}
I20260620 02:52:36.265623 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:36.260576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56190 (local address 127.0.97.3:35045)
0620 02:52:36.262075 (+ 1499us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:36.262086 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:36.262123 (+ 37us) server_negotiation.cc:407] Connection header received
0620 02:52:36.262192 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:36.262199 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:36.262451 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:36.262615 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:36.264657 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:36.264665 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:36.264680 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:52:36.264698 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:36.264761 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:52:36.264782 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:36.264955 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:36.265058 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 02:52:36.265159 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1297,"thread_start_us":1141,"threads_started":1}
I20260620 02:52:36.266307 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:36.259289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45649 (local address 127.0.0.1:42474)
0620 02:52:36.259815 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:36.259840 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:36.259953 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:36.262814 (+ 2861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:36.262820 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:36.262848 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:36.263617 (+ 769us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:52:36.263621 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:36.263641 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:52:36.263694 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:36.264532 (+ 838us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:36.264537 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:36.264542 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 02:52:36.264666 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 02:52:36.264706 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":462,"threads_started":1}
I20260620 02:52:36.266980 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:36.259694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42474 (local address 127.0.97.2:45649)
0620 02:52:36.262286 (+ 2592us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:36.262302 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:36.262328 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:52:36.262444 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:36.262453 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:36.262647 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:36.262796 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:36.263901 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:36.263912 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:36.263930 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:52:36.263963 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:36.264341 (+ 378us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:52:36.264391 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:36.266460 (+ 2069us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:36.266544 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 02:52:36.266680 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2437,"thread_start_us":2716,"threads_started":1}
I20260620 02:52:36.268507 5185 tablet_service.cc:1467] Processing CreateTablet for tablet 8fd0f47bc6e44e82b901008959774fb2 (DEFAULT_TABLE table=test-table [id=561f631245c5439d9dbc514acd220c7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:36.269999 5185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fd0f47bc6e44e82b901008959774fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:36.269876 5446 tablet_service.cc:1467] Processing CreateTablet for tablet 8fd0f47bc6e44e82b901008959774fb2 (DEFAULT_TABLE table=test-table [id=561f631245c5439d9dbc514acd220c7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:36.270532 5316 tablet_service.cc:1467] Processing CreateTablet for tablet 8fd0f47bc6e44e82b901008959774fb2 (DEFAULT_TABLE table=test-table [id=561f631245c5439d9dbc514acd220c7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:36.271453 5446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fd0f47bc6e44e82b901008959774fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:36.271931 5316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fd0f47bc6e44e82b901008959774fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:36.282549 5529 tablet_bootstrap.cc:492] T 8fd0f47bc6e44e82b901008959774fb2 P 041d670e51c74fc9a970667ff4fa67d1: Bootstrap starting.
I20260620 02:52:36.286255 5529 tablet_bootstrap.cc:654] T 8fd0f47bc6e44e82b901008959774fb2 P 041d670e51c74fc9a970667ff4fa67d1: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:36.286478 5530 tablet_bootstrap.cc:492] T 8fd0f47bc6e44e82b901008959774fb2 P b2197c40b54b49bc9f47cedb2991c7b5: Bootstrap starting.
I20260620 02:52:36.287848 5529 log.cc:824] T 8fd0f47bc6e44e82b901008959774fb2 P 041d670e51c74fc9a970667ff4fa67d1: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:36.291021 5530 tablet_bootstrap.cc:654] T 8fd0f47bc6e44e82b901008959774fb2 P b2197c40b54b49bc9f47cedb2991c7b5: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:36.292681 5530 log.cc:824] T 8fd0f47bc6e44e82b901008959774fb2 P b2197c40b54b49bc9f47cedb2991c7b5: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:36.293377 5531 tablet_bootstrap.cc:492] T 8fd0f47bc6e44e82b901008959774fb2 P a17c3994f2944e9297649ca936d551bb: Bootstrap starting.
I20260620 02:52:36.298177 5531 tablet_bootstrap.cc:654] T 8fd0f47bc6e44e82b901008959774fb2 P a17c3994f2944e9297649ca936d551bb: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:36.300557 5531 log.cc:824] T 8fd0f47bc6e44e82b901008959774fb2 P a17c3994f2944e9297649ca936d551bb: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:36.301247 5529 tablet_bootstrap.cc:492] T 8fd0f47bc6e44e82b901008959774fb2 P 041d670e51c74fc9a970667ff4fa67d1: No bootstrap required, opened a new log
I20260620 02:52:36.301537 5529 ts_tablet_manager.cc:1397] T 8fd0f47bc6e44e82b901008959774fb2 P 041d670e51c74fc9a970667ff4fa67d1: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.012s
I20260620 02:52:36.301637 5530 tablet_bootstrap.cc:492] T 8fd0f47bc6e44e82b901008959774fb2 P b2197c40b54b49bc9f47cedb2991c7b5: No bootstrap required, opened a new log
I20260620 02:52:36.302027 5530 ts_tablet_manager.cc:1397] T 8fd0f47bc6e44e82b901008959774fb2 P b2197c40b54b49bc9f47cedb2991c7b5: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.005s
I20260620 02:52:36.308815 5250 heartbeater.cc:503] Master 127.0.97.62:35501 was elected leader, sending a full tablet report...
I20260620 02:52:36.309852 5531 tablet_bootstrap.cc:492] T 8fd0f47bc6e44e82b901008959774fb2 P a17c3994f2944e9297649ca936d551bb: No bootstrap required, opened a new log
I20260620 02:52:36.310205 5531 ts_tablet_manager.cc:1397] T 8fd0f47bc6e44e82b901008959774fb2 P a17c3994f2944e9297649ca936d551bb: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20260620 02:52:36.313256 5529 raft_consensus.cc:348] T 8fd0f47bc6e44e82b901008959774fb2 P 041d670e51c74fc9a970667ff4fa67d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d670e51c74fc9a970667ff4fa67d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34957 } } peers { permanent_uuid: "a17c3994f2944e9297649ca936d551bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45649 } } peers { permanent_uuid: "b2197c40b54b49bc9f47cedb2991c7b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35045 } }
I20260620 02:52:36.313673 5529 raft_consensus.cc:374] T 8fd0f47bc6e44e82b901008959774fb2 P 041d670e51c74fc9a970667ff4fa67d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:36.313803 5529 raft_consensus.cc:729] T 8fd0f47bc6e44e82b901008959774fb2 P 041d670e51c74fc9a970667ff4fa67d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 041d670e51c74fc9a970667ff4fa67d1, State: Initialized, Role: FOLLOWER
I20260620 02:52:36.314186 5530 raft_consensus.cc:348] T 8fd0f47bc6e44e82b901008959774fb2 P b2197c40b54b49bc9f47cedb2991c7b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d670e51c74fc9a970667ff4fa67d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34957 } } peers { permanent_uuid: "a17c3994f2944e9297649ca936d551bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45649 } } peers { permanent_uuid: "b2197c40b54b49bc9f47cedb2991c7b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35045 } }
I20260620 02:52:36.314682 5530 raft_consensus.cc:374] T 8fd0f47bc6e44e82b901008959774fb2 P b2197c40b54b49bc9f47cedb2991c7b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:36.314517 5529 consensus_queue.cc:260] T 8fd0f47bc6e44e82b901008959774fb2 P 041d670e51c74fc9a970667ff4fa67d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d670e51c74fc9a970667ff4fa67d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34957 } } peers { permanent_uuid: "a17c3994f2944e9297649ca936d551bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45649 } } peers { permanent_uuid: "b2197c40b54b49bc9f47cedb2991c7b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35045 } }
I20260620 02:52:36.314782 5530 raft_consensus.cc:729] T 8fd0f47bc6e44e82b901008959774fb2 P b2197c40b54b49bc9f47cedb2991c7b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2197c40b54b49bc9f47cedb2991c7b5, State: Initialized, Role: FOLLOWER
I20260620 02:52:36.315533 5530 consensus_queue.cc:260] T 8fd0f47bc6e44e82b901008959774fb2 P b2197c40b54b49bc9f47cedb2991c7b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d670e51c74fc9a970667ff4fa67d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34957 } } peers { permanent_uuid: "a17c3994f2944e9297649ca936d551bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45649 } } peers { permanent_uuid: "b2197c40b54b49bc9f47cedb2991c7b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35045 } }
I20260620 02:52:36.316412 5529 ts_tablet_manager.cc:1428] T 8fd0f47bc6e44e82b901008959774fb2 P 041d670e51c74fc9a970667ff4fa67d1: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260620 02:52:36.319082 5511 heartbeater.cc:503] Master 127.0.97.62:35501 was elected leader, sending a full tablet report...
I20260620 02:52:36.319751 5530 ts_tablet_manager.cc:1428] T 8fd0f47bc6e44e82b901008959774fb2 P b2197c40b54b49bc9f47cedb2991c7b5: Time spent starting tablet: real 0.018s user 0.013s sys 0.003s
I20260620 02:52:36.321566 5531 raft_consensus.cc:348] T 8fd0f47bc6e44e82b901008959774fb2 P a17c3994f2944e9297649ca936d551bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d670e51c74fc9a970667ff4fa67d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34957 } } peers { permanent_uuid: "a17c3994f2944e9297649ca936d551bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45649 } } peers { permanent_uuid: "b2197c40b54b49bc9f47cedb2991c7b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35045 } }
I20260620 02:52:36.321985 5531 raft_consensus.cc:374] T 8fd0f47bc6e44e82b901008959774fb2 P a17c3994f2944e9297649ca936d551bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:36.322104 5531 raft_consensus.cc:729] T 8fd0f47bc6e44e82b901008959774fb2 P a17c3994f2944e9297649ca936d551bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a17c3994f2944e9297649ca936d551bb, State: Initialized, Role: FOLLOWER
I20260620 02:52:36.322708 5531 consensus_queue.cc:260] T 8fd0f47bc6e44e82b901008959774fb2 P a17c3994f2944e9297649ca936d551bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d670e51c74fc9a970667ff4fa67d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34957 } } peers { permanent_uuid: "a17c3994f2944e9297649ca936d551bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45649 } } peers { permanent_uuid: "b2197c40b54b49bc9f47cedb2991c7b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35045 } }
I20260620 02:52:36.323725 5381 heartbeater.cc:503] Master 127.0.97.62:35501 was elected leader, sending a full tablet report...
I20260620 02:52:36.324323 5531 ts_tablet_manager.cc:1428] T 8fd0f47bc6e44e82b901008959774fb2 P a17c3994f2944e9297649ca936d551bb: Time spent starting tablet: real 0.014s user 0.004s sys 0.009s
I20260620 02:52:36.437161 5535 raft_consensus.cc:482] T 8fd0f47bc6e44e82b901008959774fb2 P 041d670e51c74fc9a970667ff4fa67d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:52:36.437538 5535 raft_consensus.cc:504] T 8fd0f47bc6e44e82b901008959774fb2 P 041d670e51c74fc9a970667ff4fa67d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d670e51c74fc9a970667ff4fa67d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34957 } } peers { permanent_uuid: "a17c3994f2944e9297649ca936d551bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45649 } } peers { permanent_uuid: "b2197c40b54b49bc9f47cedb2991c7b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35045 } }
I20260620 02:52:36.439334 5535 leader_election.cc:288] T 8fd0f47bc6e44e82b901008959774fb2 P 041d670e51c74fc9a970667ff4fa67d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a17c3994f2944e9297649ca936d551bb (127.0.97.2:45649), b2197c40b54b49bc9f47cedb2991c7b5 (127.0.97.3:35045)
I20260620 02:52:36.442771 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:36.439877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59021 (local address 127.0.97.2:45649)
0620 02:52:36.440101 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:36.440110 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:36.440653 (+ 543us) server_negotiation.cc:407] Connection header received
0620 02:52:36.440837 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:36.440842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:36.440933 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:36.441043 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:36.442092 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:36.442098 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:36.442103 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:52:36.442120 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:36.442159 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:52:36.442171 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:36.442284 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:36.442433 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 02:52:36.442495 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:52:36.442986 5538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:36.439790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45649 (local address 127.0.97.1:59021)
0620 02:52:36.440529 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:36.440545 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:36.440708 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:36.441068 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:36.441073 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:36.441099 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:36.441894 (+ 795us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:52:36.441897 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:36.441910 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:52:36.441950 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:36.442270 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:36.442275 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:36.442279 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 02:52:36.442500 (+ 221us) client_negotiation.cc:240] Negotiation successful
0620 02:52:36.442550 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":445,"threads_started":1}
I20260620 02:52:36.443889 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:36.440966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40757 (local address 127.0.97.3:35045)
0620 02:52:36.441209 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:36.441218 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:36.441875 (+ 657us) server_negotiation.cc:407] Connection header received
0620 02:52:36.442062 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:36.442067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:36.442117 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:36.442229 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:36.443294 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:36.443299 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:36.443302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:52:36.443316 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:36.443347 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:52:36.443358 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:36.443524 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:36.443643 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 02:52:36.443701 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 02:52:36.444255 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:36.440656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35045 (local address 127.0.97.1:40757)
0620 02:52:36.441758 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:36.441772 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:36.441916 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:36.442248 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:36.442252 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:36.442278 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:36.443100 (+ 822us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:52:36.443103 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:36.443123 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:52:36.443166 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:36.443536 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:36.443539 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:36.443543 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 02:52:36.443667 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 02:52:36.443741 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1002,"thread_start_us":809,"threads_started":1}
I20260620 02:52:36.444288 5336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fd0f47bc6e44e82b901008959774fb2" candidate_uuid: "041d670e51c74fc9a970667ff4fa67d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a17c3994f2944e9297649ca936d551bb" is_pre_election: true
I20260620 02:52:36.444867 5336 raft_consensus.cc:2454] T 8fd0f47bc6e44e82b901008959774fb2 P a17c3994f2944e9297649ca936d551bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 041d670e51c74fc9a970667ff4fa67d1 in term 0.
I20260620 02:52:36.445474 5466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fd0f47bc6e44e82b901008959774fb2" candidate_uuid: "041d670e51c74fc9a970667ff4fa67d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2197c40b54b49bc9f47cedb2991c7b5" is_pre_election: true
I20260620 02:52:36.445611 5141 leader_election.cc:302] T 8fd0f47bc6e44e82b901008959774fb2 P 041d670e51c74fc9a970667ff4fa67d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 041d670e51c74fc9a970667ff4fa67d1, a17c3994f2944e9297649ca936d551bb; no voters:
I20260620 02:52:36.446040 5535 raft_consensus.cc:2781] T 8fd0f47bc6e44e82b901008959774fb2 P 041d670e51c74fc9a970667ff4fa67d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:52:36.446013 5466 raft_consensus.cc:2454] T 8fd0f47bc6e44e82b901008959774fb2 P b2197c40b54b49bc9f47cedb2991c7b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 041d670e51c74fc9a970667ff4fa67d1 in term 0.
I20260620 02:52:36.446189 5535 raft_consensus.cc:482] T 8fd0f47bc6e44e82b901008959774fb2 P 041d670e51c74fc9a970667ff4fa67d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:52:36.446313 5535 raft_consensus.cc:3037] T 8fd0f47bc6e44e82b901008959774fb2 P 041d670e51c74fc9a970667ff4fa67d1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:36.448559 5535 raft_consensus.cc:504] T 8fd0f47bc6e44e82b901008959774fb2 P 041d670e51c74fc9a970667ff4fa67d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d670e51c74fc9a970667ff4fa67d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34957 } } peers { permanent_uuid: "a17c3994f2944e9297649ca936d551bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45649 } } peers { permanent_uuid: "b2197c40b54b49bc9f47cedb2991c7b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35045 } }
I20260620 02:52:36.449252 5535 leader_election.cc:288] T 8fd0f47bc6e44e82b901008959774fb2 P 041d670e51c74fc9a970667ff4fa67d1 [CANDIDATE]: Term 1 election: Requested vote from peers a17c3994f2944e9297649ca936d551bb (127.0.97.2:45649), b2197c40b54b49bc9f47cedb2991c7b5 (127.0.97.3:35045)
I20260620 02:52:36.450057 5336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fd0f47bc6e44e82b901008959774fb2" candidate_uuid: "041d670e51c74fc9a970667ff4fa67d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a17c3994f2944e9297649ca936d551bb"
I20260620 02:52:36.450121 5466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fd0f47bc6e44e82b901008959774fb2" candidate_uuid: "041d670e51c74fc9a970667ff4fa67d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2197c40b54b49bc9f47cedb2991c7b5"
I20260620 02:52:36.450277 5336 raft_consensus.cc:3037] T 8fd0f47bc6e44e82b901008959774fb2 P a17c3994f2944e9297649ca936d551bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:36.450299 5466 raft_consensus.cc:3037] T 8fd0f47bc6e44e82b901008959774fb2 P b2197c40b54b49bc9f47cedb2991c7b5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:36.453550 5466 raft_consensus.cc:2454] T 8fd0f47bc6e44e82b901008959774fb2 P b2197c40b54b49bc9f47cedb2991c7b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 041d670e51c74fc9a970667ff4fa67d1 in term 1.
I20260620 02:52:36.453560 5336 raft_consensus.cc:2454] T 8fd0f47bc6e44e82b901008959774fb2 P a17c3994f2944e9297649ca936d551bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 041d670e51c74fc9a970667ff4fa67d1 in term 1.
I20260620 02:52:36.453945 5141 leader_election.cc:302] T 8fd0f47bc6e44e82b901008959774fb2 P 041d670e51c74fc9a970667ff4fa67d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 041d670e51c74fc9a970667ff4fa67d1, a17c3994f2944e9297649ca936d551bb; no voters:
I20260620 02:52:36.454216 5535 raft_consensus.cc:2781] T 8fd0f47bc6e44e82b901008959774fb2 P 041d670e51c74fc9a970667ff4fa67d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:36.454733 5535 raft_consensus.cc:686] T 8fd0f47bc6e44e82b901008959774fb2 P 041d670e51c74fc9a970667ff4fa67d1 [term 1 LEADER]: Becoming Leader. State: Replica: 041d670e51c74fc9a970667ff4fa67d1, State: Running, Role: LEADER
I20260620 02:52:36.455262 5535 consensus_queue.cc:237] T 8fd0f47bc6e44e82b901008959774fb2 P 041d670e51c74fc9a970667ff4fa67d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d670e51c74fc9a970667ff4fa67d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34957 } } peers { permanent_uuid: "a17c3994f2944e9297649ca936d551bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45649 } } peers { permanent_uuid: "b2197c40b54b49bc9f47cedb2991c7b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35045 } }
W20260620 02:52:36.459456 5512 tablet.cc:2369] T 8fd0f47bc6e44e82b901008959774fb2 P b2197c40b54b49bc9f47cedb2991c7b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:52:36.460014 5064 catalog_manager.cc:5495] T 8fd0f47bc6e44e82b901008959774fb2 P 041d670e51c74fc9a970667ff4fa67d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 041d670e51c74fc9a970667ff4fa67d1 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "041d670e51c74fc9a970667ff4fa67d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041d670e51c74fc9a970667ff4fa67d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a17c3994f2944e9297649ca936d551bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2197c40b54b49bc9f47cedb2991c7b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35045 } health_report { overall_health: UNKNOWN } } }
W20260620 02:52:36.499956 5382 tablet.cc:2369] T 8fd0f47bc6e44e82b901008959774fb2 P a17c3994f2944e9297649ca936d551bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:52:36.506081 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:36.503787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59884 (local address 127.0.97.1:34957)
0620 02:52:36.504006 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:36.504011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:36.504030 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:52:36.504122 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:36.504126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:36.504213 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:36.504307 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:36.505342 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:36.505347 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:36.505351 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:52:36.505369 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:36.505405 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:52:36.505418 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:36.505537 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:36.505648 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 02:52:36.505700 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 02:52:36.515087 5336 raft_consensus.cc:1261] T 8fd0f47bc6e44e82b901008959774fb2 P a17c3994f2944e9297649ca936d551bb [term 1 FOLLOWER]: Refusing update from remote peer 041d670e51c74fc9a970667ff4fa67d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:36.515087 5466 raft_consensus.cc:1261] T 8fd0f47bc6e44e82b901008959774fb2 P b2197c40b54b49bc9f47cedb2991c7b5 [term 1 FOLLOWER]: Refusing update from remote peer 041d670e51c74fc9a970667ff4fa67d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:36.516149 5540 consensus_queue.cc:1035] T 8fd0f47bc6e44e82b901008959774fb2 P 041d670e51c74fc9a970667ff4fa67d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17c3994f2944e9297649ca936d551bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:52:36.516538 5535 consensus_queue.cc:1035] T 8fd0f47bc6e44e82b901008959774fb2 P 041d670e51c74fc9a970667ff4fa67d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2197c40b54b49bc9f47cedb2991c7b5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:52:36.527530 5543 mvcc.cc:205] Tried to move back new op lower bound from 7298760525874331648 to 7298760525644206080. Current Snapshot: MvccSnapshot[applied={T|T < 7298760525874331648}]
I20260620 02:52:36.541798 5064 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:52:36.542105 5064 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:52:36.545966 5064 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f48e06309e6843ad8d88979564a35bb9: Sending DeleteTablet for 3 replicas of tablet 8fd0f47bc6e44e82b901008959774fb2
I20260620 02:52:36.547334 5185 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fd0f47bc6e44e82b901008959774fb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:36 UTC) from {username='slave'} at 127.0.0.1:59872
I20260620 02:52:36.547506 5446 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fd0f47bc6e44e82b901008959774fb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:36 UTC) from {username='slave'} at 127.0.0.1:56190
I20260620 02:52:36.548030 5316 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fd0f47bc6e44e82b901008959774fb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:36 UTC) from {username='slave'} at 127.0.0.1:42474
I20260620 02:52:36.550182 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 5123
I20260620 02:52:36.562780 5552 tablet_replica.cc:330] T 8fd0f47bc6e44e82b901008959774fb2 P 041d670e51c74fc9a970667ff4fa67d1: stopping tablet replica
I20260620 02:52:36.563401 5552 raft_consensus.cc:2229] T 8fd0f47bc6e44e82b901008959774fb2 P 041d670e51c74fc9a970667ff4fa67d1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:52:36.564181 5552 raft_consensus.cc:2258] T 8fd0f47bc6e44e82b901008959774fb2 P 041d670e51c74fc9a970667ff4fa67d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:36.566877 5552 ts_tablet_manager.cc:1905] T 8fd0f47bc6e44e82b901008959774fb2 P 041d670e51c74fc9a970667ff4fa67d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:36.575544 5552 ts_tablet_manager.cc:1918] T 8fd0f47bc6e44e82b901008959774fb2 P 041d670e51c74fc9a970667ff4fa67d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:52:36.576097 5552 log.cc:1196] T 8fd0f47bc6e44e82b901008959774fb2 P 041d670e51c74fc9a970667ff4fa67d1: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-0/wal/wals/8fd0f47bc6e44e82b901008959774fb2
I20260620 02:52:36.576164 5554 tablet_replica.cc:330] T 8fd0f47bc6e44e82b901008959774fb2 P a17c3994f2944e9297649ca936d551bb: stopping tablet replica
I20260620 02:52:36.576663 5552 ts_tablet_manager.cc:1939] T 8fd0f47bc6e44e82b901008959774fb2 P 041d670e51c74fc9a970667ff4fa67d1: Deleting consensus metadata
I20260620 02:52:36.576679 5554 raft_consensus.cc:2229] T 8fd0f47bc6e44e82b901008959774fb2 P a17c3994f2944e9297649ca936d551bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:52:36.577183 5554 raft_consensus.cc:2258] T 8fd0f47bc6e44e82b901008959774fb2 P a17c3994f2944e9297649ca936d551bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:36.578555 5051 catalog_manager.cc:4841] TS 041d670e51c74fc9a970667ff4fa67d1 (127.0.97.1:34957): tablet 8fd0f47bc6e44e82b901008959774fb2 (table test-table [id=561f631245c5439d9dbc514acd220c7a]) successfully deleted
I20260620 02:52:36.578768 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:36.564251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59886 (local address 127.0.97.1:34957)
0620 02:52:36.564775 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:36.564782 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:36.574354 (+ 9572us) server_negotiation.cc:407] Connection header received
0620 02:52:36.574880 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:36.574886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:36.574947 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:36.575013 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:36.577913 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:36.577918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:36.577923 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:52:36.577942 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:36.577977 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:52:36.577992 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:36.578088 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:36.578382 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 02:52:36.578451 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378}
I20260620 02:52:36.580018 5245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:36.580623 5553 tablet_replica.cc:330] T 8fd0f47bc6e44e82b901008959774fb2 P b2197c40b54b49bc9f47cedb2991c7b5: stopping tablet replica
I20260620 02:52:36.581102 5553 raft_consensus.cc:2229] T 8fd0f47bc6e44e82b901008959774fb2 P b2197c40b54b49bc9f47cedb2991c7b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:52:36.581681 5553 raft_consensus.cc:2258] T 8fd0f47bc6e44e82b901008959774fb2 P b2197c40b54b49bc9f47cedb2991c7b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:36.581792 5554 ts_tablet_manager.cc:1905] T 8fd0f47bc6e44e82b901008959774fb2 P a17c3994f2944e9297649ca936d551bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:36.584245 5553 ts_tablet_manager.cc:1905] T 8fd0f47bc6e44e82b901008959774fb2 P b2197c40b54b49bc9f47cedb2991c7b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:36.588606 5554 ts_tablet_manager.cc:1918] T 8fd0f47bc6e44e82b901008959774fb2 P a17c3994f2944e9297649ca936d551bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:52:36.588774 5554 log.cc:1196] T 8fd0f47bc6e44e82b901008959774fb2 P a17c3994f2944e9297649ca936d551bb: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-1/wal/wals/8fd0f47bc6e44e82b901008959774fb2
I20260620 02:52:36.589257 5554 ts_tablet_manager.cc:1939] T 8fd0f47bc6e44e82b901008959774fb2 P a17c3994f2944e9297649ca936d551bb: Deleting consensus metadata
I20260620 02:52:36.591113 5553 ts_tablet_manager.cc:1918] T 8fd0f47bc6e44e82b901008959774fb2 P b2197c40b54b49bc9f47cedb2991c7b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:52:36.591163 5051 catalog_manager.cc:4841] TS a17c3994f2944e9297649ca936d551bb (127.0.97.2:45649): tablet 8fd0f47bc6e44e82b901008959774fb2 (table test-table [id=561f631245c5439d9dbc514acd220c7a]) successfully deleted
I20260620 02:52:36.591284 5553 log.cc:1196] T 8fd0f47bc6e44e82b901008959774fb2 P b2197c40b54b49bc9f47cedb2991c7b5: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781923914418765-388-0/minicluster-data/ts-2/wal/wals/8fd0f47bc6e44e82b901008959774fb2
I20260620 02:52:36.591699 5553 ts_tablet_manager.cc:1939] T 8fd0f47bc6e44e82b901008959774fb2 P b2197c40b54b49bc9f47cedb2991c7b5: Deleting consensus metadata
I20260620 02:52:36.593097 5051 catalog_manager.cc:4841] TS b2197c40b54b49bc9f47cedb2991c7b5 (127.0.97.3:35045): tablet 8fd0f47bc6e44e82b901008959774fb2 (table test-table [id=561f631245c5439d9dbc514acd220c7a]) successfully deleted
I20260620 02:52:36.677196 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 5123
I20260620 02:52:36.694952 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 5254
I20260620 02:52:36.697945 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:36.695759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42480 (local address 127.0.97.2:45649)
0620 02:52:36.696001 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:36.696006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:36.696027 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:52:36.696097 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:36.696102 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:36.696194 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:36.696315 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:36.697241 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:36.697247 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:36.697251 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:52:36.697269 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:36.697307 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:52:36.697319 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:36.697445 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:36.697607 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 02:52:36.697660 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:52:36.698694 5376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:36.791452 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 5254
I20260620 02:52:36.809734 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 5384
I20260620 02:52:36.813881 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:36.810709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56206 (local address 127.0.97.3:35045)
0620 02:52:36.810915 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:36.810924 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:36.810943 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:52:36.811215 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:36.811219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:36.811307 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:36.811415 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:36.812643 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:36.812648 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:36.812652 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:52:36.812933 (+ 281us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:36.812976 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:52:36.812996 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:36.813127 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:36.813563 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 02:52:36.813635 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260620 02:52:36.814651 5506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:36.908144 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 5384
I20260620 02:52:36.926018 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 5033
I20260620 02:52:36.927268 5094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:37.039198 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 5033
2026-06-20T02:52:37Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2750 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 02:52:37.080137 388 mini_oidc.cc:152] Starting JWKS server
I20260620 02:52:37.082677 388 webserver.cc:460] Webserver started at https://127.0.0.1:34963/ using document root <none> and password file <none>
I20260620 02:52:37.082939 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 02:52:37.083671 388 webserver.cc:460] Webserver started at http://127.0.0.1:39441/ using document root <none> and password file <none>
2026-06-20T02:52:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:52:37Z Disabled control of system clock
I20260620 02:52:37.109421 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39051
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39051
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34963/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781923914418765-388-0/testchainca.cert with env {}
W20260620 02:52:37.424242 5565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:37.424532 5565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:37.424701 5565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:37.432026 5565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:52:37.432134 5565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:37.432209 5565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:37.432307 5565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:52:37.432392 5565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 02:52:37.432474 5565 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 02:52:37.432541 5565 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34963/jwks.json
I20260620 02:52:37.442981 5565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39051
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34963/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781923914418765-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:37.444247 5565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:37.445585 5565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:37.457551 5570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:37.457625 5571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:37.457864 5573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:37.458432 5565 server_base.cc:1034] running on GCE node
I20260620 02:52:37.459224 5565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:37.460650 5565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:37.461860 5565 hybrid_clock.cc:648] HybridClock initialized: now 1781923957461774 us; error 84 us; skew 500 ppm
I20260620 02:52:37.462239 5565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:52:37.464586 5565 webserver.cc:460] Webserver started at http://127.0.97.62:41389/ using document root <none> and password file <none>
I20260620 02:52:37.465258 5565 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:37.465348 5565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:37.465682 5565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:37.468644 5565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "4874f9830eac4a95a5962c4ecc50eeca"
format_stamp: "Formatted at 2026-06-20 02:52:37 on dist-test-slave-8p8c"
server_key: "9c13c7b2cd1ded7e6e68ee410e0e41a5"
server_key_iv: "6f995ef11a3eca93a8676170fd6ce97b"
server_key_version: "encryptionkey@0"
I20260620 02:52:37.469458 5565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "4874f9830eac4a95a5962c4ecc50eeca"
format_stamp: "Formatted at 2026-06-20 02:52:37 on dist-test-slave-8p8c"
server_key: "9c13c7b2cd1ded7e6e68ee410e0e41a5"
server_key_iv: "6f995ef11a3eca93a8676170fd6ce97b"
server_key_version: "encryptionkey@0"
I20260620 02:52:37.474535 5565 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 02:52:37.478081 5579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:37.479593 5565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:52:37.479768 5565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "4874f9830eac4a95a5962c4ecc50eeca"
format_stamp: "Formatted at 2026-06-20 02:52:37 on dist-test-slave-8p8c"
server_key: "9c13c7b2cd1ded7e6e68ee410e0e41a5"
server_key_iv: "6f995ef11a3eca93a8676170fd6ce97b"
server_key_version: "encryptionkey@0"
I20260620 02:52:37.479974 5565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:37.491804 5565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:37.492584 5565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:37.492853 5565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:37.511405 5565 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39051
I20260620 02:52:37.511438 5630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39051 every 8 connection(s)
I20260620 02:52:37.512737 5565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:52:37.516590 5631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:37.517050 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 5565
I20260620 02:52:37.517261 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:52:37.518107 388 external_mini_cluster.cc:1428] Setting key b639ed98e737c7544442c46b24246b8f
I20260620 02:52:37.531670 5631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4874f9830eac4a95a5962c4ecc50eeca: Bootstrap starting.
I20260620 02:52:37.533459 5634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:37.520183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45716 (local address 127.0.97.62:39051)
0620 02:52:37.521296 (+ 1113us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:37.521312 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:37.521359 (+ 47us) server_negotiation.cc:407] Connection header received
0620 02:52:37.523192 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:37.523218 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:37.523743 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:37.524166 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:37.524896 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:37.526259 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:37.528621 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:37.529329 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:37.529471 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:37.530828 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:37.530836 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:37.530855 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:52:37.531465 (+ 610us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:37.531544 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:52:37.531565 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:37.531664 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:37.531972 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 02:52:37.532361 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":380,"threads_started":1}
I20260620 02:52:37.535130 5631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4874f9830eac4a95a5962c4ecc50eeca: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:37.536216 5631 log.cc:824] T 00000000000000000000000000000000 P 4874f9830eac4a95a5962c4ecc50eeca: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:37.537721 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:52:37.539376 5631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4874f9830eac4a95a5962c4ecc50eeca: No bootstrap required, opened a new log
I20260620 02:52:37.547605 5631 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4874f9830eac4a95a5962c4ecc50eeca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4874f9830eac4a95a5962c4ecc50eeca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39051 } }
I20260620 02:52:37.548040 5631 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4874f9830eac4a95a5962c4ecc50eeca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:37.548158 5631 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4874f9830eac4a95a5962c4ecc50eeca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4874f9830eac4a95a5962c4ecc50eeca, State: Initialized, Role: FOLLOWER
I20260620 02:52:37.548782 5631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4874f9830eac4a95a5962c4ecc50eeca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4874f9830eac4a95a5962c4ecc50eeca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39051 } }
I20260620 02:52:37.549039 5631 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4874f9830eac4a95a5962c4ecc50eeca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:52:37.549188 5631 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4874f9830eac4a95a5962c4ecc50eeca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:52:37.549367 5631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4874f9830eac4a95a5962c4ecc50eeca [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:37.551837 5631 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4874f9830eac4a95a5962c4ecc50eeca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4874f9830eac4a95a5962c4ecc50eeca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39051 } }
I20260620 02:52:37.552412 5631 leader_election.cc:302] T 00000000000000000000000000000000 P 4874f9830eac4a95a5962c4ecc50eeca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4874f9830eac4a95a5962c4ecc50eeca; no voters:
I20260620 02:52:37.553083 5631 leader_election.cc:288] T 00000000000000000000000000000000 P 4874f9830eac4a95a5962c4ecc50eeca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:52:37.553246 5636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4874f9830eac4a95a5962c4ecc50eeca [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:37.554824 5636 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4874f9830eac4a95a5962c4ecc50eeca [term 1 LEADER]: Becoming Leader. State: Replica: 4874f9830eac4a95a5962c4ecc50eeca, State: Running, Role: LEADER
I20260620 02:52:37.555388 5636 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4874f9830eac4a95a5962c4ecc50eeca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4874f9830eac4a95a5962c4ecc50eeca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39051 } }
I20260620 02:52:37.555882 5631 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4874f9830eac4a95a5962c4ecc50eeca [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:52:37.558669 5637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4874f9830eac4a95a5962c4ecc50eeca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4874f9830eac4a95a5962c4ecc50eeca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4874f9830eac4a95a5962c4ecc50eeca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39051 } } }
I20260620 02:52:37.559104 5637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4874f9830eac4a95a5962c4ecc50eeca [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:37.559656 5638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4874f9830eac4a95a5962c4ecc50eeca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4874f9830eac4a95a5962c4ecc50eeca. Latest consensus state: current_term: 1 leader_uuid: "4874f9830eac4a95a5962c4ecc50eeca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4874f9830eac4a95a5962c4ecc50eeca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39051 } } }
I20260620 02:52:37.560103 5638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4874f9830eac4a95a5962c4ecc50eeca [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:37.561551 5645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:52:37.566713 5645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:52:37.576089 5645 catalog_manager.cc:1347] Generated new cluster ID: 70590050c09348cd8692c44d6cde341f
I20260620 02:52:37.576246 5645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:52:37.599057 5645 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:52:37.600361 5645 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:52:37.609789 5645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4874f9830eac4a95a5962c4ecc50eeca: Generated new TSK 0
I20260620 02:52:37.610716 5645 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 02:52:37.662659 5661 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 02:52:37.662853 5661 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 02:52:37.663636 5634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:37.635164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45718 (local address 127.0.97.62:39051)
0620 02:52:37.635414 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:37.635421 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:37.635868 (+ 447us) server_negotiation.cc:407] Connection header received
0620 02:52:37.636282 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:37.636286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:37.636361 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:37.636516 (+ 155us) server_negotiation.cc:226] Negotiated authn=JWT
0620 02:52:37.637228 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:37.637975 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:37.640356 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:37.640814 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:37.640917 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:37.641615 (+ 698us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 02:52:37.663125 (+ 21510us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34963/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 02:52:37.663409 (+ 284us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45718: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34963/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 02:52:37.663792 5634 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45718: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34963/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 02:52:37.665742 5660 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:52:37.635051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39051 (local address 127.0.0.1:45718)
0620 02:52:37.635744 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:37.635766 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:37.635967 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:37.636583 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:37.636589 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:37.636604 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0620 02:52:37.637054 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:37.637068 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:37.638123 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:37.638130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:37.640028 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:37.640045 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:37.640939 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:37.641010 (+ 71us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:37.641214 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:37.641291 (+ 77us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 02:52:37.663440 (+ 22149us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34963/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 02:52:37.663824 (+ 384us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:39051: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34963/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":334,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:39051: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34963/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 02:52:37.667969 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 5565
I20260620 02:52:37.669214 5626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:37.766110 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 5565
2026-06-20T02:52:37Z chronyd exiting
I20260620 02:52:37.816748 388 test_util.cc:175] -----------------------------------------------
I20260620 02:52:37.816920 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781923914418765-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (741 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 02:52:37.820644 388 mini_oidc.cc:152] Starting JWKS server
I20260620 02:52:37.822441 388 webserver.cc:460] Webserver started at https://127.0.0.1:44731/ using document root <none> and password file <none>
I20260620 02:52:37.822683 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 02:52:37.823323 388 webserver.cc:460] Webserver started at http://127.0.0.1:44615/ using document root <none> and password file <none>
2026-06-20T02:52:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:52:37Z Disabled control of system clock
I20260620 02:52:37.849534 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42899
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42899
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44731/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781923914418765-388-0/testchainca.cert with env {}
W20260620 02:52:38.184725 5668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:38.185078 5668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:38.185225 5668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:38.192631 5668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:52:38.192776 5668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:38.192862 5668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:38.192961 5668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:52:38.193039 5668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 02:52:38.193125 5668 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 02:52:38.193195 5668 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44731/jwks.json
I20260620 02:52:38.203425 5668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42899
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44731/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781923914418765-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:38.204792 5668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:38.206136 5668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:38.218567 5676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:38.218761 5674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:38.219228 5673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:38.220629 5668 server_base.cc:1034] running on GCE node
I20260620 02:52:38.221402 5668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:38.222877 5668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:38.224113 5668 hybrid_clock.cc:648] HybridClock initialized: now 1781923958224058 us; error 39 us; skew 500 ppm
I20260620 02:52:38.224562 5668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:52:38.232364 5668 webserver.cc:460] Webserver started at http://127.0.97.62:46629/ using document root <none> and password file <none>
I20260620 02:52:38.233114 5668 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:38.233211 5668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:38.233570 5668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:38.236590 5668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "140cf3814fc341b7b93af94e8582fff9"
format_stamp: "Formatted at 2026-06-20 02:52:38 on dist-test-slave-8p8c"
server_key: "48e93637aeb64d0ef0220acc262a1bbe"
server_key_iv: "ba69364af2a7747dca93c3cff0828136"
server_key_version: "encryptionkey@0"
I20260620 02:52:38.237396 5668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "140cf3814fc341b7b93af94e8582fff9"
format_stamp: "Formatted at 2026-06-20 02:52:38 on dist-test-slave-8p8c"
server_key: "48e93637aeb64d0ef0220acc262a1bbe"
server_key_iv: "ba69364af2a7747dca93c3cff0828136"
server_key_version: "encryptionkey@0"
I20260620 02:52:38.242449 5668 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 02:52:38.245997 5682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:38.247526 5668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:52:38.247714 5668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "140cf3814fc341b7b93af94e8582fff9"
format_stamp: "Formatted at 2026-06-20 02:52:38 on dist-test-slave-8p8c"
server_key: "48e93637aeb64d0ef0220acc262a1bbe"
server_key_iv: "ba69364af2a7747dca93c3cff0828136"
server_key_version: "encryptionkey@0"
I20260620 02:52:38.247925 5668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:38.263864 5668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:38.264675 5668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:38.264945 5668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:38.283233 5668 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42899
I20260620 02:52:38.283252 5733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42899 every 8 connection(s)
I20260620 02:52:38.284544 5668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:52:38.288311 5734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:38.289475 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 5668
I20260620 02:52:38.289657 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:52:38.290273 388 external_mini_cluster.cc:1428] Setting key 62c31c1d849c6724da0820e60c003194
I20260620 02:52:38.300931 5734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 140cf3814fc341b7b93af94e8582fff9: Bootstrap starting.
I20260620 02:52:38.303200 5737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:38.292153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50892 (local address 127.0.97.62:42899)
0620 02:52:38.293310 (+ 1157us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:38.293321 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:38.293372 (+ 51us) server_negotiation.cc:407] Connection header received
0620 02:52:38.295021 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:38.295051 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:38.295522 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:38.295931 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:38.296460 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:38.297687 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:38.298970 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:38.299626 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:38.299819 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:38.300942 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:38.300949 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:38.300963 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:52:38.301526 (+ 563us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:38.301604 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:52:38.301624 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:38.301748 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:38.301946 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 02:52:38.302240 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":500,"threads_started":1}
I20260620 02:52:38.304404 5734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 140cf3814fc341b7b93af94e8582fff9: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:38.305500 5734 log.cc:824] T 00000000000000000000000000000000 P 140cf3814fc341b7b93af94e8582fff9: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:38.306847 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:52:38.308139 5734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 140cf3814fc341b7b93af94e8582fff9: No bootstrap required, opened a new log
I20260620 02:52:38.314977 5734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 140cf3814fc341b7b93af94e8582fff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "140cf3814fc341b7b93af94e8582fff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42899 } }
I20260620 02:52:38.315325 5734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 140cf3814fc341b7b93af94e8582fff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:38.315425 5734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 140cf3814fc341b7b93af94e8582fff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 140cf3814fc341b7b93af94e8582fff9, State: Initialized, Role: FOLLOWER
I20260620 02:52:38.315943 5734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 140cf3814fc341b7b93af94e8582fff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "140cf3814fc341b7b93af94e8582fff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42899 } }
I20260620 02:52:38.316164 5734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 140cf3814fc341b7b93af94e8582fff9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:52:38.316296 5734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 140cf3814fc341b7b93af94e8582fff9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:52:38.316452 5734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 140cf3814fc341b7b93af94e8582fff9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:38.318511 5734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 140cf3814fc341b7b93af94e8582fff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "140cf3814fc341b7b93af94e8582fff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42899 } }
I20260620 02:52:38.318979 5734 leader_election.cc:302] T 00000000000000000000000000000000 P 140cf3814fc341b7b93af94e8582fff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 140cf3814fc341b7b93af94e8582fff9; no voters:
I20260620 02:52:38.319607 5734 leader_election.cc:288] T 00000000000000000000000000000000 P 140cf3814fc341b7b93af94e8582fff9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:52:38.319798 5739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 140cf3814fc341b7b93af94e8582fff9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:38.320595 5739 raft_consensus.cc:686] T 00000000000000000000000000000000 P 140cf3814fc341b7b93af94e8582fff9 [term 1 LEADER]: Becoming Leader. State: Replica: 140cf3814fc341b7b93af94e8582fff9, State: Running, Role: LEADER
I20260620 02:52:38.321128 5739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 140cf3814fc341b7b93af94e8582fff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "140cf3814fc341b7b93af94e8582fff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42899 } }
I20260620 02:52:38.321691 5734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 140cf3814fc341b7b93af94e8582fff9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:52:38.323802 5740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 140cf3814fc341b7b93af94e8582fff9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "140cf3814fc341b7b93af94e8582fff9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "140cf3814fc341b7b93af94e8582fff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42899 } } }
I20260620 02:52:38.324242 5740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 140cf3814fc341b7b93af94e8582fff9 [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:38.325829 5748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:52:38.331158 5748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:52:38.334335 5741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 140cf3814fc341b7b93af94e8582fff9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 140cf3814fc341b7b93af94e8582fff9. Latest consensus state: current_term: 1 leader_uuid: "140cf3814fc341b7b93af94e8582fff9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "140cf3814fc341b7b93af94e8582fff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42899 } } }
I20260620 02:52:38.334869 5741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 140cf3814fc341b7b93af94e8582fff9 [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:38.340416 5748 catalog_manager.cc:1347] Generated new cluster ID: 1c8f9f9cd3784d9f90ff5e8966583c0e
I20260620 02:52:38.340519 5748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:52:38.358673 5748 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:52:38.359930 5748 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:52:38.367230 5748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 140cf3814fc341b7b93af94e8582fff9: Generated new TSK 0
I20260620 02:52:38.368054 5748 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 02:52:38.403726 5764 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 02:52:38.403872 5764 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 02:52:38.405130 5737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:38.384470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50896 (local address 127.0.97.62:42899)
0620 02:52:38.384658 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:38.384665 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:38.384966 (+ 301us) server_negotiation.cc:407] Connection header received
0620 02:52:38.385169 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:38.385175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:38.385276 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:38.385440 (+ 164us) server_negotiation.cc:226] Negotiated authn=JWT
0620 02:52:38.385854 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:38.387059 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:38.388548 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:38.389241 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:38.389372 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:38.390020 (+ 648us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 02:52:38.404412 (+ 14392us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44731/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 02:52:38.404794 (+ 382us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50896: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44731/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 02:52:38.405167 5763 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:52:38.384302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42899 (local address 127.0.0.1:50896)
0620 02:52:38.384867 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:38.384886 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:38.385008 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:38.385451 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:38.385455 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:38.385467 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0620 02:52:38.385702 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:38.385709 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:38.387224 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:38.387229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:38.388398 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:38.388410 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:38.389489 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:38.389541 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:38.389702 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:38.389737 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 02:52:38.404670 (+ 14933us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44731/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 02:52:38.404880 (+ 210us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:42899: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44731/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":321,"threads_started":1}
W20260620 02:52:38.405257 5737 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50896: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44731/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260620 02:52:38.406352 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 5668
I20260620 02:52:38.407577 5729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:38.502310 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 5668
2026-06-20T02:52:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (736 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 02:52:38.557494 388 mini_oidc.cc:152] Starting JWKS server
I20260620 02:52:38.559419 388 webserver.cc:460] Webserver started at https://127.0.0.1:32821/ using document root <none> and password file <none>
I20260620 02:52:38.559669 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 02:52:38.560277 388 webserver.cc:460] Webserver started at http://127.0.0.1:40803/ using document root <none> and password file <none>
2026-06-20T02:52:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:52:38Z Disabled control of system clock
I20260620 02:52:38.586229 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41493
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41493
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32821/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 02:52:38.925061 5771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:38.925376 5771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:38.925465 5771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:38.933120 5771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:52:38.933250 5771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:38.933288 5771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:38.933341 5771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:52:38.933385 5771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 02:52:38.933440 5771 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 02:52:38.933480 5771 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:32821/jwks.json
I20260620 02:52:38.942956 5771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41493
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32821/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:38.944346 5771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:38.946208 5771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:38.958841 5776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:38.958868 5777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:38.959012 5779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:38.960469 5771 server_base.cc:1034] running on GCE node
I20260620 02:52:38.961365 5771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:38.962972 5771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:38.964232 5771 hybrid_clock.cc:648] HybridClock initialized: now 1781923958964177 us; error 39 us; skew 500 ppm
I20260620 02:52:38.964609 5771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:52:38.967119 5771 webserver.cc:460] Webserver started at http://127.0.97.62:35925/ using document root <none> and password file <none>
I20260620 02:52:38.967845 5771 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:38.967978 5771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:38.968317 5771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:38.971446 5771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "58379b906e2847ccb544a6fea0effbe3"
format_stamp: "Formatted at 2026-06-20 02:52:38 on dist-test-slave-8p8c"
server_key: "c5377392bfcef3272809ee45b49304c2"
server_key_iv: "28d5056ad5eab1c4ef92e38f65fcd6a5"
server_key_version: "encryptionkey@0"
I20260620 02:52:38.972306 5771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "58379b906e2847ccb544a6fea0effbe3"
format_stamp: "Formatted at 2026-06-20 02:52:38 on dist-test-slave-8p8c"
server_key: "c5377392bfcef3272809ee45b49304c2"
server_key_iv: "28d5056ad5eab1c4ef92e38f65fcd6a5"
server_key_version: "encryptionkey@0"
I20260620 02:52:38.977941 5771 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:52:38.982277 5785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:38.983971 5771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260620 02:52:38.984189 5771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "58379b906e2847ccb544a6fea0effbe3"
format_stamp: "Formatted at 2026-06-20 02:52:38 on dist-test-slave-8p8c"
server_key: "c5377392bfcef3272809ee45b49304c2"
server_key_iv: "28d5056ad5eab1c4ef92e38f65fcd6a5"
server_key_version: "encryptionkey@0"
I20260620 02:52:38.984417 5771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:39.001368 5771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:39.002245 5771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:39.002561 5771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:39.021552 5771 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41493
I20260620 02:52:39.021595 5836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41493 every 8 connection(s)
I20260620 02:52:39.022957 5771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:52:39.025511 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 5771
I20260620 02:52:39.025767 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:52:39.026402 388 external_mini_cluster.cc:1428] Setting key ef1d59b895e4d90d0223c46f9eb92ee8
I20260620 02:52:39.027019 5837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:39.041896 5837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58379b906e2847ccb544a6fea0effbe3: Bootstrap starting.
I20260620 02:52:39.042552 5840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:39.028228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40110 (local address 127.0.97.62:41493)
0620 02:52:39.029333 (+ 1105us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:39.029349 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:39.029394 (+ 45us) server_negotiation.cc:407] Connection header received
0620 02:52:39.031548 (+ 2154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:39.031571 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:39.032084 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:39.032435 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:39.033454 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:39.035279 (+ 1825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:39.037299 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:39.038063 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:39.038216 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:39.039493 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:39.039501 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:39.039519 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:52:39.040146 (+ 627us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:39.040242 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:52:39.040265 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:39.040362 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:39.040642 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 02:52:39.041068 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":389,"threads_started":1}
I20260620 02:52:39.045390 5837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58379b906e2847ccb544a6fea0effbe3: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:39.046572 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:52:39.046790 5837 log.cc:824] T 00000000000000000000000000000000 P 58379b906e2847ccb544a6fea0effbe3: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:39.050020 5837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58379b906e2847ccb544a6fea0effbe3: No bootstrap required, opened a new log
I20260620 02:52:39.057242 5837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58379b906e2847ccb544a6fea0effbe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58379b906e2847ccb544a6fea0effbe3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41493 } }
I20260620 02:52:39.057651 5837 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58379b906e2847ccb544a6fea0effbe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:39.057751 5837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58379b906e2847ccb544a6fea0effbe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58379b906e2847ccb544a6fea0effbe3, State: Initialized, Role: FOLLOWER
I20260620 02:52:39.058497 5837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58379b906e2847ccb544a6fea0effbe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58379b906e2847ccb544a6fea0effbe3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41493 } }
I20260620 02:52:39.058735 5837 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58379b906e2847ccb544a6fea0effbe3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:52:39.058996 5837 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58379b906e2847ccb544a6fea0effbe3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:52:39.059288 5837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58379b906e2847ccb544a6fea0effbe3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:39.061960 5837 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58379b906e2847ccb544a6fea0effbe3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58379b906e2847ccb544a6fea0effbe3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41493 } }
I20260620 02:52:39.062646 5837 leader_election.cc:302] T 00000000000000000000000000000000 P 58379b906e2847ccb544a6fea0effbe3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58379b906e2847ccb544a6fea0effbe3; no voters:
I20260620 02:52:39.063321 5837 leader_election.cc:288] T 00000000000000000000000000000000 P 58379b906e2847ccb544a6fea0effbe3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:52:39.063541 5842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58379b906e2847ccb544a6fea0effbe3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:39.064239 5842 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58379b906e2847ccb544a6fea0effbe3 [term 1 LEADER]: Becoming Leader. State: Replica: 58379b906e2847ccb544a6fea0effbe3, State: Running, Role: LEADER
I20260620 02:52:39.064929 5842 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58379b906e2847ccb544a6fea0effbe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58379b906e2847ccb544a6fea0effbe3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41493 } }
I20260620 02:52:39.065711 5837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58379b906e2847ccb544a6fea0effbe3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:52:39.068212 5843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58379b906e2847ccb544a6fea0effbe3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58379b906e2847ccb544a6fea0effbe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58379b906e2847ccb544a6fea0effbe3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41493 } } }
I20260620 02:52:39.068719 5843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58379b906e2847ccb544a6fea0effbe3 [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:39.068301 5844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58379b906e2847ccb544a6fea0effbe3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58379b906e2847ccb544a6fea0effbe3. Latest consensus state: current_term: 1 leader_uuid: "58379b906e2847ccb544a6fea0effbe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58379b906e2847ccb544a6fea0effbe3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41493 } } }
I20260620 02:52:39.069675 5844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58379b906e2847ccb544a6fea0effbe3 [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:39.070763 5851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:52:39.077050 5851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:52:39.087023 5851 catalog_manager.cc:1347] Generated new cluster ID: 373bbed594674712b01de94ab536d731
I20260620 02:52:39.087178 5851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:52:39.099344 5851 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:52:39.101217 5851 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:52:39.127060 5851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58379b906e2847ccb544a6fea0effbe3: Generated new TSK 0
I20260620 02:52:39.128187 5851 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:52:39.156082 5840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:39.146711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40114 (local address 127.0.97.62:41493)
0620 02:52:39.146920 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:39.146926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:39.147467 (+ 541us) server_negotiation.cc:407] Connection header received
0620 02:52:39.147694 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:39.147701 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:39.147799 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:39.147977 (+ 178us) server_negotiation.cc:226] Negotiated authn=JWT
0620 02:52:39.148498 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:39.149663 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:39.151332 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:39.152018 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:39.152188 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:39.152900 (+ 712us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 02:52:39.155409 (+ 2509us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32821/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 02:52:39.155756 (+ 347us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40114: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32821/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 02:52:39.156219 5840 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40114: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32821/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 02:52:39.157838 5866 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:52:39.146576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41493 (local address 127.0.0.1:40114)
0620 02:52:39.147352 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:39.147371 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:39.147506 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:39.148026 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:39.148029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:39.148039 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0620 02:52:39.148345 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:39.148354 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:39.149831 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:39.149835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:39.151196 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:39.151209 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:39.152225 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:39.152308 (+ 83us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:39.152520 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:39.152571 (+ 51us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 02:52:39.155641 (+ 3070us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32821/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 02:52:39.157416 (+ 1775us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:41493: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32821/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":1463,"thread_start_us":439,"threads_started":1}
I20260620 02:52:39.160531 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 5771
I20260620 02:52:39.161926 5832 generic_service.cc:196] Checking for leaks (request via RPC)
E20260620 02:52:39.169931 5867 webserver.cc:533] Webserver: sslize failed:
E20260620 02:52:39.170086 5867 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 02:52:39.258179 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 5771
2026-06-20T02:52:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (757 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T02:52:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:52:39Z Disabled control of system clock
I20260620 02:52:39.341306 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42387
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42387
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781923914418765-388-0/testchainca.cert with env {}
W20260620 02:52:39.661911 5872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:39.662241 5872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:39.662416 5872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:39.670090 5872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:52:39.670214 5872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:39.670285 5872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:39.670418 5872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:52:39.670491 5872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 02:52:39.670601 5872 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 02:52:39.670662 5872 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 02:52:39.681078 5872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42387
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781923914418765-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:39.682448 5872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:39.683750 5872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:39.695900 5880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:39.696019 5878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:39.696178 5877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:39.697130 5872 server_base.cc:1034] running on GCE node
I20260620 02:52:39.697940 5872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:39.699417 5872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:39.700618 5872 hybrid_clock.cc:648] HybridClock initialized: now 1781923959700564 us; error 55 us; skew 500 ppm
I20260620 02:52:39.700929 5872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:52:39.703367 5872 webserver.cc:460] Webserver started at http://127.0.97.62:43443/ using document root <none> and password file <none>
I20260620 02:52:39.704020 5872 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:39.704107 5872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:39.704360 5872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:39.707248 5872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "a0fae77fd3ba4544a0acf405e579538a"
format_stamp: "Formatted at 2026-06-20 02:52:39 on dist-test-slave-8p8c"
server_key: "7262317d7fbb97490e144e2a451ae889"
server_key_iv: "3cbc6fd02422d9867d2a2f5eb863d3c6"
server_key_version: "encryptionkey@0"
I20260620 02:52:39.707965 5872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "a0fae77fd3ba4544a0acf405e579538a"
format_stamp: "Formatted at 2026-06-20 02:52:39 on dist-test-slave-8p8c"
server_key: "7262317d7fbb97490e144e2a451ae889"
server_key_iv: "3cbc6fd02422d9867d2a2f5eb863d3c6"
server_key_version: "encryptionkey@0"
I20260620 02:52:39.713001 5872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:52:39.716742 5886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:39.718304 5872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:52:39.718525 5872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "a0fae77fd3ba4544a0acf405e579538a"
format_stamp: "Formatted at 2026-06-20 02:52:39 on dist-test-slave-8p8c"
server_key: "7262317d7fbb97490e144e2a451ae889"
server_key_iv: "3cbc6fd02422d9867d2a2f5eb863d3c6"
server_key_version: "encryptionkey@0"
I20260620 02:52:39.718714 5872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:39.738631 5872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:39.739447 5872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:39.739689 5872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:39.757515 5872 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42387
I20260620 02:52:39.757539 5937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42387 every 8 connection(s)
I20260620 02:52:39.758908 5872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:52:39.761018 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 5872
I20260620 02:52:39.761212 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:52:39.761925 388 external_mini_cluster.cc:1428] Setting key 58481b575591bd63243e64006f30c2a3
I20260620 02:52:39.762995 5938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:39.776331 5938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0fae77fd3ba4544a0acf405e579538a: Bootstrap starting.
I20260620 02:52:39.776757 5941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:39.763804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58826 (local address 127.0.97.62:42387)
0620 02:52:39.764738 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:39.764749 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:39.764789 (+ 40us) server_negotiation.cc:407] Connection header received
0620 02:52:39.767248 (+ 2459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:39.767271 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:39.767785 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:39.768122 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:39.768797 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:39.770700 (+ 1903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:39.772354 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:39.773018 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:39.773240 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:39.774485 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:39.774493 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:39.774507 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:52:39.775046 (+ 539us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:39.775123 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:52:39.775143 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:39.775262 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:39.775486 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 02:52:39.775786 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":338,"threads_started":1}
I20260620 02:52:39.779980 5938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0fae77fd3ba4544a0acf405e579538a: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:39.780613 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:52:39.781080 5938 log.cc:824] T 00000000000000000000000000000000 P a0fae77fd3ba4544a0acf405e579538a: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:39.783885 5938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0fae77fd3ba4544a0acf405e579538a: No bootstrap required, opened a new log
I20260620 02:52:39.790853 5938 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0fae77fd3ba4544a0acf405e579538a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0fae77fd3ba4544a0acf405e579538a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42387 } }
I20260620 02:52:39.791219 5938 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0fae77fd3ba4544a0acf405e579538a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:39.791322 5938 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0fae77fd3ba4544a0acf405e579538a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0fae77fd3ba4544a0acf405e579538a, State: Initialized, Role: FOLLOWER
I20260620 02:52:39.791833 5938 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0fae77fd3ba4544a0acf405e579538a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0fae77fd3ba4544a0acf405e579538a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42387 } }
I20260620 02:52:39.792062 5938 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0fae77fd3ba4544a0acf405e579538a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:52:39.792196 5938 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0fae77fd3ba4544a0acf405e579538a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:52:39.792353 5938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0fae77fd3ba4544a0acf405e579538a [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:39.794551 5938 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0fae77fd3ba4544a0acf405e579538a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0fae77fd3ba4544a0acf405e579538a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42387 } }
I20260620 02:52:39.795032 5938 leader_election.cc:302] T 00000000000000000000000000000000 P a0fae77fd3ba4544a0acf405e579538a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0fae77fd3ba4544a0acf405e579538a; no voters:
I20260620 02:52:39.795614 5938 leader_election.cc:288] T 00000000000000000000000000000000 P a0fae77fd3ba4544a0acf405e579538a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:52:39.795760 5943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0fae77fd3ba4544a0acf405e579538a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:39.796556 5943 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0fae77fd3ba4544a0acf405e579538a [term 1 LEADER]: Becoming Leader. State: Replica: a0fae77fd3ba4544a0acf405e579538a, State: Running, Role: LEADER
I20260620 02:52:39.797081 5943 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0fae77fd3ba4544a0acf405e579538a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0fae77fd3ba4544a0acf405e579538a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42387 } }
I20260620 02:52:39.797701 5938 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0fae77fd3ba4544a0acf405e579538a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:52:39.800249 5945 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0fae77fd3ba4544a0acf405e579538a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0fae77fd3ba4544a0acf405e579538a. Latest consensus state: current_term: 1 leader_uuid: "a0fae77fd3ba4544a0acf405e579538a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0fae77fd3ba4544a0acf405e579538a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42387 } } }
I20260620 02:52:39.800343 5944 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0fae77fd3ba4544a0acf405e579538a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0fae77fd3ba4544a0acf405e579538a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0fae77fd3ba4544a0acf405e579538a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42387 } } }
I20260620 02:52:39.800587 5945 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0fae77fd3ba4544a0acf405e579538a [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:39.800695 5944 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0fae77fd3ba4544a0acf405e579538a [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:39.802536 5950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:52:39.808068 5950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:52:39.817127 5950 catalog_manager.cc:1347] Generated new cluster ID: 6287f389a1014039bd163a005221e79f
I20260620 02:52:39.817287 5950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:52:39.844065 5950 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:52:39.845352 5950 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:52:39.853240 5950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0fae77fd3ba4544a0acf405e579538a: Generated new TSK 0
I20260620 02:52:39.854121 5950 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:52:39.966745 5941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:39.870642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58834 (local address 127.0.97.62:42387)
0620 02:52:39.870818 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:39.870826 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:39.871404 (+ 578us) server_negotiation.cc:407] Connection header received
0620 02:52:39.871628 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:39.871632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:39.871702 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:39.871841 (+ 139us) server_negotiation.cc:226] Negotiated authn=JWT
0620 02:52:39.872393 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:39.873291 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:39.875398 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:39.875839 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:39.875954 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:39.876549 (+ 595us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 02:52:39.966076 (+ 89527us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 02:52:39.966477 (+ 401us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58834: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 02:52:39.966904 5941 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58834: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 02:52:39.967113 5967 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:52:39.870451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42387 (local address 127.0.0.1:58834)
0620 02:52:39.871235 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:39.871255 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:39.871410 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:39.871881 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:39.871886 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:39.871900 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0620 02:52:39.872191 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:39.872202 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:39.873456 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:39.873462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:39.875252 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:39.875268 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:39.875957 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:39.876010 (+ 53us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:39.876205 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:39.876251 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 02:52:39.966393 (+ 90142us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 02:52:39.966714 (+ 321us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:42387: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":475,"threads_started":1}
I20260620 02:52:39.969110 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 5872
I20260620 02:52:39.970463 5933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:40.073808 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 5872
2026-06-20T02:52:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (793 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (410 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (406 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:52:40 dist-test-slave-8p8c krb5kdc[5982](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:52:40 dist-test-slave-8p8c krb5kdc[5982](info): set up 2 sockets
Jun 20 02:52:40 dist-test-slave-8p8c krb5kdc[5982](info): commencing operation
krb5kdc: starting...
W20260620 02:52:41.991178 388 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:52:42 dist-test-slave-8p8c krb5kdc[5982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:52:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:52:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:52:42.194875 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45475
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:42.519713 5995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:42.520069 5995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:42.520224 5995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:42.527381 5995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:52:42.527534 5995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:42.527642 5995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:42.527746 5995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:52:42.527827 5995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:52:42.537647 5995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:42.539007 5995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:42.540326 5995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:42.551849 6003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:42.551911 6001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:42.552085 6000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:42.552685 5995 server_base.cc:1034] running on GCE node
I20260620 02:52:42.553519 5995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:42.555385 5995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:42.556613 5995 hybrid_clock.cc:648] HybridClock initialized: now 1781923962556562 us; error 44 us; skew 500 ppm
I20260620 02:52:42.557015 5995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:42 dist-test-slave-8p8c krb5kdc[5982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:42.560300 5995 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:52:42.561827 5995 webserver.cc:460] Webserver started at http://127.0.97.62:42107/ using document root <none> and password file <none>
I20260620 02:52:42.562724 5995 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:42.562861 5995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:42.563177 5995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:42.566154 5995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "65ef82122ef04f96913841a0bf4ac516"
format_stamp: "Formatted at 2026-06-20 02:52:42 on dist-test-slave-8p8c"
server_key: "2247a1d1747190a1b30ae079e85cc9c5"
server_key_iv: "d93cc15103495135e193e0e61749214a"
server_key_version: "encryptionkey@0"
I20260620 02:52:42.566995 5995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "65ef82122ef04f96913841a0bf4ac516"
format_stamp: "Formatted at 2026-06-20 02:52:42 on dist-test-slave-8p8c"
server_key: "2247a1d1747190a1b30ae079e85cc9c5"
server_key_iv: "d93cc15103495135e193e0e61749214a"
server_key_version: "encryptionkey@0"
I20260620 02:52:42.572184 5995 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:52:42.575783 6010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:42.577185 5995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:52:42.577375 5995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "65ef82122ef04f96913841a0bf4ac516"
format_stamp: "Formatted at 2026-06-20 02:52:42 on dist-test-slave-8p8c"
server_key: "2247a1d1747190a1b30ae079e85cc9c5"
server_key_iv: "d93cc15103495135e193e0e61749214a"
server_key_version: "encryptionkey@0"
I20260620 02:52:42.577577 5995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:42.593814 5995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:42.598482 5995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:42.598771 5995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:42.617785 5995 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45475
I20260620 02:52:42.617813 6061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45475 every 8 connection(s)
I20260620 02:52:42.619210 5995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:52:42.623008 6062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:42.623935 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 5995
I20260620 02:52:42.624153 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:52:42.624773 388 external_mini_cluster.cc:1428] Setting key 086d8bfb5e5bba8b9920ca53c276e3ef
I20260620 02:52:42.636814 6062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ef82122ef04f96913841a0bf4ac516: Bootstrap starting.
Jun 20 02:52:42 dist-test-slave-8p8c krb5kdc[5982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:42.640597 6062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65ef82122ef04f96913841a0bf4ac516: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:42.641973 6062 log.cc:824] T 00000000000000000000000000000000 P 65ef82122ef04f96913841a0bf4ac516: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:42.645089 6062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ef82122ef04f96913841a0bf4ac516: No bootstrap required, opened a new log
I20260620 02:52:42.648906 6065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:42.626766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59950 (local address 127.0.97.62:45475)
0620 02:52:42.627579 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:42.627591 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:42.627634 (+ 43us) server_negotiation.cc:407] Connection header received
0620 02:52:42.629187 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:42.629212 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:42.629771 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:42.630184 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:42.631839 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:42.633204 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:42.635042 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:42.635790 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:42.635935 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:42.640511 (+ 4576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:42.640523 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:42.640542 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:42.640599 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:42.644336 (+ 3737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:42.645164 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:42.645172 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:42.645179 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:42.645374 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:42.645747 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:42.645754 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:42.645760 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:42.646460 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:42.646749 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:42.647096 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 02:52:42.647557 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":374,"threads_started":1}
I20260620 02:52:42.654134 6062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65ef82122ef04f96913841a0bf4ac516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ef82122ef04f96913841a0bf4ac516" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45475 } }
I20260620 02:52:42.654495 6062 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65ef82122ef04f96913841a0bf4ac516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:42.654570 6062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65ef82122ef04f96913841a0bf4ac516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65ef82122ef04f96913841a0bf4ac516, State: Initialized, Role: FOLLOWER
I20260620 02:52:42.655026 6062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65ef82122ef04f96913841a0bf4ac516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ef82122ef04f96913841a0bf4ac516" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45475 } }
I20260620 02:52:42.655226 6062 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65ef82122ef04f96913841a0bf4ac516 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:52:42.655365 6062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65ef82122ef04f96913841a0bf4ac516 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:52:42.655524 6062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65ef82122ef04f96913841a0bf4ac516 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:42.657706 6062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65ef82122ef04f96913841a0bf4ac516 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ef82122ef04f96913841a0bf4ac516" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45475 } }
I20260620 02:52:42.658169 6062 leader_election.cc:302] T 00000000000000000000000000000000 P 65ef82122ef04f96913841a0bf4ac516 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65ef82122ef04f96913841a0bf4ac516; no voters:
I20260620 02:52:42.658766 6062 leader_election.cc:288] T 00000000000000000000000000000000 P 65ef82122ef04f96913841a0bf4ac516 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:52:42.658898 6067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65ef82122ef04f96913841a0bf4ac516 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:42.659622 6067 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65ef82122ef04f96913841a0bf4ac516 [term 1 LEADER]: Becoming Leader. State: Replica: 65ef82122ef04f96913841a0bf4ac516, State: Running, Role: LEADER
I20260620 02:52:42.660238 6067 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65ef82122ef04f96913841a0bf4ac516 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ef82122ef04f96913841a0bf4ac516" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45475 } }
I20260620 02:52:42.660923 6062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65ef82122ef04f96913841a0bf4ac516 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:52:42.663529 6069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ef82122ef04f96913841a0bf4ac516 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65ef82122ef04f96913841a0bf4ac516" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ef82122ef04f96913841a0bf4ac516" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45475 } } }
I20260620 02:52:42.663611 6068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ef82122ef04f96913841a0bf4ac516 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65ef82122ef04f96913841a0bf4ac516. Latest consensus state: current_term: 1 leader_uuid: "65ef82122ef04f96913841a0bf4ac516" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ef82122ef04f96913841a0bf4ac516" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45475 } } }
I20260620 02:52:42.664063 6069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ef82122ef04f96913841a0bf4ac516 [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:42.664121 6068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ef82122ef04f96913841a0bf4ac516 [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:42.667850 6073 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:52:42.673352 6073 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:52:42.682232 6073 catalog_manager.cc:1347] Generated new cluster ID: 76b9280e8eea46659a83a53d923c17f5
I20260620 02:52:42.682345 6073 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:52:42.726346 6073 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:52:42.727761 6073 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:52:42.732944 6073 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65ef82122ef04f96913841a0bf4ac516: Generated new TSK 0
I20260620 02:52:42.733731 6073 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:52:42.782524 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45475
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:43.126945 6088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:43.127250 6088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:43.127342 6088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:43.134681 6088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:43.134796 6088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:43.134910 6088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:52:43.144770 6088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:43.146028 6088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:43.147714 6088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:43.163465 6094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:43.163723 6093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:43.164079 6096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:43.165288 6088 server_base.cc:1034] running on GCE node
I20260620 02:52:43.166105 6088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:43.167487 6088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:43.168752 6088 hybrid_clock.cc:648] HybridClock initialized: now 1781923963168675 us; error 71 us; skew 500 ppm
I20260620 02:52:43.169168 6088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:43 dist-test-slave-8p8c krb5kdc[5982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:43.172739 6088 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:52:43.174321 6088 webserver.cc:460] Webserver started at http://127.0.97.1:45333/ using document root <none> and password file <none>
I20260620 02:52:43.175082 6088 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:43.175207 6088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:43.175518 6088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:43.178536 6088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-0/data/instance:
uuid: "4bbb67ce47be440d825cc8868f6d0638"
format_stamp: "Formatted at 2026-06-20 02:52:43 on dist-test-slave-8p8c"
server_key: "c826e9fba4c011ded15042c132be4018"
server_key_iv: "129fb7080bf195e79fe10fdfd1c53d95"
server_key_version: "encryptionkey@0"
I20260620 02:52:43.179347 6088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4bbb67ce47be440d825cc8868f6d0638"
format_stamp: "Formatted at 2026-06-20 02:52:43 on dist-test-slave-8p8c"
server_key: "c826e9fba4c011ded15042c132be4018"
server_key_iv: "129fb7080bf195e79fe10fdfd1c53d95"
server_key_version: "encryptionkey@0"
I20260620 02:52:43.185161 6088 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:52:43.189105 6103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:43.190675 6088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 02:52:43.190816 6088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-0/wal
uuid: "4bbb67ce47be440d825cc8868f6d0638"
format_stamp: "Formatted at 2026-06-20 02:52:43 on dist-test-slave-8p8c"
server_key: "c826e9fba4c011ded15042c132be4018"
server_key_iv: "129fb7080bf195e79fe10fdfd1c53d95"
server_key_version: "encryptionkey@0"
I20260620 02:52:43.191005 6088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:43.219117 6088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:43.223625 6088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:43.223883 6088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:43.225019 6088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:52:43.226470 6088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:43.226544 6088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:43.226640 6088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:43.226683 6088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:43.261843 6088 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38971
I20260620 02:52:43.261938 6215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38971 every 8 connection(s)
I20260620 02:52:43.263396 6088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:52:43.273372 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 6088
I20260620 02:52:43.273594 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:52:43.274389 388 external_mini_cluster.cc:1428] Setting key e20cc3d18eea3bf4fb7a68eb18946a32
Jun 20 02:52:43 dist-test-slave-8p8c krb5kdc[5982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:43.289714 6218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:43.266113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57975 (local address 127.0.97.62:45475)
0620 02:52:43.266662 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:43.266667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:43.267738 (+ 1071us) server_negotiation.cc:407] Connection header received
0620 02:52:43.269903 (+ 2165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:43.269907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:43.269974 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:43.270111 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:43.271745 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:43.272566 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:43.275064 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:43.275689 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:43.275824 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:43.283721 (+ 7897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:43.283726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:43.283729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:43.283761 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:43.286412 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:43.287365 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:43.287369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:43.287371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:43.287490 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:43.287958 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:43.287961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:43.287962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:43.288255 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:43.288361 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:43.289281 (+ 920us) server_negotiation.cc:299] Negotiation successful
0620 02:52:43.289451 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":282,"threads_started":1}
I20260620 02:52:43.291316 6219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:43.266737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45475 (local address 127.0.97.1:57975)
0620 02:52:43.267543 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:43.267608 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:43.269624 (+ 2016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:43.270336 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:43.270344 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:43.270935 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:43.271561 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:43.271578 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:43.272727 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:43.272731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:43.274846 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:43.274866 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:43.275912 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:43.275917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:43.276167 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:43.280137 (+ 3970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:43.280173 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:43.283457 (+ 3284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:43.286558 (+ 3101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:43.286570 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:43.286590 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:43.287229 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:43.287602 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:43.287609 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:43.287616 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:43.287852 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:43.288380 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:43.288391 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:43.288838 (+ 447us) client_negotiation.cc:769] Sending connection context
0620 02:52:43.289161 (+ 323us) client_negotiation.cc:240] Negotiation successful
0620 02:52:43.289749 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":347,"threads_started":1}
I20260620 02:52:43.292958 6216 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45475
I20260620 02:52:43.293278 6216 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:43.293995 6216 heartbeater.cc:511] Master 127.0.97.62:45475 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:52:43.296196 6027 ts_manager.cc:194] Registered new tserver with Master: 4bbb67ce47be440d825cc8868f6d0638 (127.0.97.1:38971)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 02:52:43.297873 6027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:52:43.321729 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45475
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:43.634414 6222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:43.634738 6222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:43.634904 6222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:43.642169 6222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:43.642315 6222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:43.642509 6222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:52:43.652099 6222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:43.653359 6222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:43.654697 6222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:43.667822 6227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:43.668036 6228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:43.668110 6230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:43.668968 6222 server_base.cc:1034] running on GCE node
I20260620 02:52:43.669600 6222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:43.670606 6222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:43.671855 6222 hybrid_clock.cc:648] HybridClock initialized: now 1781923963671799 us; error 48 us; skew 500 ppm
I20260620 02:52:43.672214 6222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:43 dist-test-slave-8p8c krb5kdc[5982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:43.675622 6222 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:52:43.677101 6222 webserver.cc:460] Webserver started at http://127.0.97.2:33289/ using document root <none> and password file <none>
I20260620 02:52:43.677768 6222 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:43.677870 6222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:43.678121 6222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:43.681203 6222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-1/data/instance:
uuid: "27afd167a04f4142828cb699f126c149"
format_stamp: "Formatted at 2026-06-20 02:52:43 on dist-test-slave-8p8c"
server_key: "05c1d1b08b42070c0ba2346f576b15bb"
server_key_iv: "0a3ae50d9d2dc075d78204c704b39ee1"
server_key_version: "encryptionkey@0"
I20260620 02:52:43.682026 6222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-1/wal/instance:
uuid: "27afd167a04f4142828cb699f126c149"
format_stamp: "Formatted at 2026-06-20 02:52:43 on dist-test-slave-8p8c"
server_key: "05c1d1b08b42070c0ba2346f576b15bb"
server_key_iv: "0a3ae50d9d2dc075d78204c704b39ee1"
server_key_version: "encryptionkey@0"
I20260620 02:52:43.687333 6222 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 02:52:43.690953 6237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:43.692476 6222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 02:52:43.692633 6222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-1/wal
uuid: "27afd167a04f4142828cb699f126c149"
format_stamp: "Formatted at 2026-06-20 02:52:43 on dist-test-slave-8p8c"
server_key: "05c1d1b08b42070c0ba2346f576b15bb"
server_key_iv: "0a3ae50d9d2dc075d78204c704b39ee1"
server_key_version: "encryptionkey@0"
I20260620 02:52:43.692826 6222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:43.705538 6222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:43.709851 6222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:43.710107 6222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:43.711002 6222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:52:43.712260 6222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:43.712332 6222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:43.712410 6222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:43.712453 6222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:43.745575 6222 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45945
I20260620 02:52:43.745656 6349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45945 every 8 connection(s)
I20260620 02:52:43.746963 6222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:52:43.749905 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 6222
I20260620 02:52:43.750082 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:52:43.750622 388 external_mini_cluster.cc:1428] Setting key 2febfb9aa1682d2621881e457d413f91
Jun 20 02:52:43 dist-test-slave-8p8c krb5kdc[5982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:43.769197 6218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:43.749605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37019 (local address 127.0.97.62:45475)
0620 02:52:43.749770 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:43.749774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:43.751339 (+ 1565us) server_negotiation.cc:407] Connection header received
0620 02:52:43.753369 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:43.753375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:43.753462 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:43.753564 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:43.755055 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:43.755741 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:43.757258 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:43.757797 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:43.757932 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:43.761668 (+ 3736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:43.761677 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:43.761685 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:43.761729 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:43.765444 (+ 3715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:43.766344 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:43.766353 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:43.766359 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:43.766547 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:43.767013 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:43.767022 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:43.767028 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:43.767509 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:43.767675 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:43.768459 (+ 784us) server_negotiation.cc:299] Negotiation successful
0620 02:52:43.768716 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 02:52:43.770159 6352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:43.750293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45475 (local address 127.0.97.2:37019)
0620 02:52:43.751157 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:43.751196 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:43.753118 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:43.753792 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:43.753799 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:43.754282 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:43.754895 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:43.754911 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:43.755891 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:43.755895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:43.757107 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:43.757117 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:43.758032 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:43.758037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:43.758239 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:43.759039 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:43.759064 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:43.761431 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:43.765612 (+ 4181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:43.765619 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:43.765634 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:43.766146 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:43.766669 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:43.766673 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:43.766675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:43.766879 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:43.767665 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:43.767673 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:43.768063 (+ 390us) client_negotiation.cc:769] Sending connection context
0620 02:52:43.768311 (+ 248us) client_negotiation.cc:240] Negotiation successful
0620 02:52:43.768840 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":371,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:52:43.771781 6350 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45475
I20260620 02:52:43.772156 6350 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:43.772926 6350 heartbeater.cc:511] Master 127.0.97.62:45475 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 02:52:43.774717 6027 ts_manager.cc:194] Registered new tserver with Master: 27afd167a04f4142828cb699f126c149 (127.0.97.2:45945)
I20260620 02:52:43.776224 6027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37019
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:52:43.803078 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45475
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:44.134189 6355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:44.134568 6355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:44.134739 6355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:44.141970 6355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:44.142112 6355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:44.142292 6355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:52:44.152278 6355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:44.153594 6355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:44.154994 6355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:44.169596 6361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:44.169617 6360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:44.169713 6363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:44.170572 6355 server_base.cc:1034] running on GCE node
I20260620 02:52:44.171226 6355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:44.172322 6355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:44.173570 6355 hybrid_clock.cc:648] HybridClock initialized: now 1781923964173521 us; error 33 us; skew 500 ppm
I20260620 02:52:44.173938 6355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:44 dist-test-slave-8p8c krb5kdc[5982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:44.177470 6355 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:52:44.179126 6355 webserver.cc:460] Webserver started at http://127.0.97.3:38545/ using document root <none> and password file <none>
I20260620 02:52:44.179816 6355 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:44.179942 6355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:44.180254 6355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:44.183348 6355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-2/data/instance:
uuid: "b90dff8e545d42fea86b3bb4a65f66f8"
format_stamp: "Formatted at 2026-06-20 02:52:44 on dist-test-slave-8p8c"
server_key: "bd87631a583f067fbe858805eab08dd8"
server_key_iv: "1f6a9c1b4ca11c5f12ec07c0d407ca3d"
server_key_version: "encryptionkey@0"
I20260620 02:52:44.184113 6355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b90dff8e545d42fea86b3bb4a65f66f8"
format_stamp: "Formatted at 2026-06-20 02:52:44 on dist-test-slave-8p8c"
server_key: "bd87631a583f067fbe858805eab08dd8"
server_key_iv: "1f6a9c1b4ca11c5f12ec07c0d407ca3d"
server_key_version: "encryptionkey@0"
I20260620 02:52:44.189508 6355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 02:52:44.193343 6370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:44.195188 6355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:52:44.195436 6355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-2/wal
uuid: "b90dff8e545d42fea86b3bb4a65f66f8"
format_stamp: "Formatted at 2026-06-20 02:52:44 on dist-test-slave-8p8c"
server_key: "bd87631a583f067fbe858805eab08dd8"
server_key_iv: "1f6a9c1b4ca11c5f12ec07c0d407ca3d"
server_key_version: "encryptionkey@0"
I20260620 02:52:44.195698 6355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:44.240620 6355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:44.245131 6355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:44.245455 6355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:44.246542 6355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:52:44.247946 6355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:44.248018 6355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:44.248153 6355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:44.248204 6355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:44.283416 6355 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37369
I20260620 02:52:44.283500 6482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37369 every 8 connection(s)
I20260620 02:52:44.284806 6355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:52:44.294677 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 6355
I20260620 02:52:44.294843 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:52:44.295346 388 external_mini_cluster.cc:1428] Setting key 97ad493072152c5594afa22fc09aa7f2
Jun 20 02:52:44 dist-test-slave-8p8c krb5kdc[5982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:52:44.301527 6216 heartbeater.cc:503] Master 127.0.97.62:45475 was elected leader, sending a full tablet report...
I20260620 02:52:44.305258 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:44.287650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41147 (local address 127.0.97.62:45475)
0620 02:52:44.288120 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:44.288124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:44.289062 (+ 938us) server_negotiation.cc:407] Connection header received
0620 02:52:44.290864 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:44.290870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:44.290955 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:44.291074 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:44.292486 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:44.293232 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:44.294849 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:44.295306 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:44.295440 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:44.299402 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:44.299407 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:44.299410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:44.299463 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:44.302256 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:44.303103 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:44.303109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:44.303116 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:44.303265 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:44.303656 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:44.303661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:44.303664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:44.303910 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:44.304058 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:44.304708 (+ 650us) server_negotiation.cc:299] Negotiation successful
0620 02:52:44.304906 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":227,"threads_started":1}
I20260620 02:52:44.306140 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:44.288203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45475 (local address 127.0.97.3:41147)
0620 02:52:44.288895 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:44.288932 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:44.290613 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:44.291297 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:44.291305 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:44.291791 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:44.292320 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:44.292335 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:44.293368 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:44.293378 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:44.294707 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:44.294722 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:44.295527 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:44.295534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:44.295754 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:44.296564 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:44.296596 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:44.299178 (+ 2582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:44.302416 (+ 3238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:44.302422 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:44.302433 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:44.302952 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:44.303366 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:44.303372 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:44.303376 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:44.303546 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:44.304030 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:44.304038 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:44.304353 (+ 315us) client_negotiation.cc:769] Sending connection context
0620 02:52:44.304595 (+ 242us) client_negotiation.cc:240] Negotiation successful
0620 02:52:44.305096 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":308,"threads_started":1}
I20260620 02:52:44.307657 6483 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45475
I20260620 02:52:44.307919 6483 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:44.308578 6483 heartbeater.cc:511] Master 127.0.97.62:45475 requested a full tablet report, sending...
I20260620 02:52:44.309839 6027 ts_manager.cc:194] Registered new tserver with Master: b90dff8e545d42fea86b3bb4a65f66f8 (127.0.97.3:37369)
I20260620 02:52:44.310705 6027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41147
I20260620 02:52:44.310851 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/test-admin.keytab.
Jun 20 02:52:44 dist-test-slave-8p8c krb5kdc[5982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:44.335955 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 02:52:44.340811 6491 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:52:44.338558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36438 (local address 127.0.97.1:38971)
0620 02:52:44.339186 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:44.339194 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:44.339220 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:52:44.339279 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:44.339285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:44.339501 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:44.339630 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:44.340421 (+ 791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36438: BlockingRecv error: recv got EOF from 127.0.0.1:36438 (error 108)
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":318,"threads_started":1}
W20260620 02:52:44.340894 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/krb5cc' of type FILE
Jun 20 02:52:44 dist-test-slave-8p8c krb5kdc[5982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:44.342437 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 02:52:44 dist-test-slave-8p8c krb5kdc[5982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:44.343721 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 02:52:44 dist-test-slave-8p8c krb5kdc[5982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:52:44.358923 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:44.345234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36448 (local address 127.0.97.1:38971)
0620 02:52:44.345483 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:44.345490 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:44.345755 (+ 265us) server_negotiation.cc:407] Connection header received
0620 02:52:44.345942 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:44.345947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:44.346010 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:44.346114 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:44.347198 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:44.348151 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:44.349361 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:44.349922 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:44.350070 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:44.353631 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:44.353641 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:44.353656 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:44.353693 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:44.356318 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:44.356914 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:44.356927 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:44.356929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:44.357037 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:44.357410 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:44.357415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:44.357419 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:44.357761 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:44.358067 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:44.358394 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 02:52:44.358661 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
W20260620 02:52:44.364679 6491 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:52:44.362972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36458 (local address 127.0.97.1:38971)
0620 02:52:44.363197 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:44.363203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:44.363393 (+ 190us) server_negotiation.cc:407] Connection header received
0620 02:52:44.363504 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:44.363508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:44.363569 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:44.363659 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:44.364454 (+ 795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36458: BlockingRecv error: recv got EOF from 127.0.0.1:36458 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 02:52:44.364806 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/krb5cc' of type FILE
Jun 20 02:52:44 dist-test-slave-8p8c krb5kdc[5982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:44.366251 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 02:52:44 dist-test-slave-8p8c krb5kdc[5982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:44.367571 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 02:52:44 dist-test-slave-8p8c krb5kdc[5982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:52:44.382107 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:44.368906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36460 (local address 127.0.97.1:38971)
0620 02:52:44.369127 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:44.369133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:44.369363 (+ 230us) server_negotiation.cc:407] Connection header received
0620 02:52:44.369511 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:44.369515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:44.369567 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:44.369642 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:44.370668 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:44.371459 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:44.372765 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:44.373219 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:44.373317 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:44.377061 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:44.377066 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:44.377069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:44.377099 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:44.379604 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:44.380203 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:44.380207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:44.380209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:44.380328 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:44.380714 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:44.380719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:44.380721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:44.381070 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:44.381277 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:44.381583 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 02:52:44.381792 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260620 02:52:44.386168 6491 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:52:44.384745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36470 (local address 127.0.97.1:38971)
0620 02:52:44.384908 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:44.384912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:44.385016 (+ 104us) server_negotiation.cc:407] Connection header received
0620 02:52:44.385137 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:44.385141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:44.385203 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:44.385275 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:44.385929 (+ 654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36470: BlockingRecv error: recv got EOF from 127.0.0.1:36470 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 02:52:44.386309 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781923914418765-388-0/krb5kdc/krb5cc' of type FILE
Jun 20 02:52:44 dist-test-slave-8p8c krb5kdc[5982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:44.387956 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 02:52:44 dist-test-slave-8p8c krb5kdc[5982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:44.389228 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 02:52:44 dist-test-slave-8p8c krb5kdc[5982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:52:44.403554 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:44.390605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36480 (local address 127.0.97.1:38971)
0620 02:52:44.390802 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:44.390809 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:44.390958 (+ 149us) server_negotiation.cc:407] Connection header received
0620 02:52:44.391137 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:44.391143 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:44.391213 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:44.391314 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:44.392332 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:44.393097 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:44.394427 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:44.394939 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:44.395091 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:44.398634 (+ 3543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:44.398638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:44.398642 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:44.398676 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:44.401106 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:44.401691 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:44.401701 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:44.401704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:44.401839 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:44.402205 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:44.402212 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:44.402215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:44.402513 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:44.402811 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:44.403065 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 02:52:44.403259 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 02:52:44.405205 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 6088
I20260620 02:52:44.415987 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:44.405835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36490 (local address 127.0.97.1:38971)
0620 02:52:44.405977 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:44.405980 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:44.406430 (+ 450us) server_negotiation.cc:407] Connection header received
0620 02:52:44.406632 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:44.406636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:44.406692 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:44.406777 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:44.407635 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:44.408313 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:44.409433 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:44.409827 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:44.409917 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:44.411689 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:44.411693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:44.411696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:44.411729 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:44.413910 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:44.414483 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:44.414486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:44.414488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:44.414594 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:44.414877 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:44.414880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:44.414882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:44.415132 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:44.415303 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:44.415564 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 02:52:44.415705 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 02:52:44.416572 6211 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:44.502336 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 6088
I20260620 02:52:44.518824 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 6222
Jun 20 02:52:44 dist-test-slave-8p8c krb5kdc[5982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:52:44.533939 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:44.519858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35210 (local address 127.0.97.2:45945)
0620 02:52:44.520444 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:44.520451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:44.520475 (+ 24us) server_negotiation.cc:407] Connection header received
0620 02:52:44.520534 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:44.520541 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:44.520774 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:44.520943 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:44.521906 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:44.522943 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:44.524142 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:44.524686 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:44.524813 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:44.528597 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:44.528609 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:44.528623 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:44.528657 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:44.531299 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:44.531978 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:44.531985 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:44.531987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:44.532109 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:44.532455 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:44.532462 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:44.532465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:44.532783 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:44.533094 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:44.533385 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 02:52:44.533633 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":286,"threads_started":1}
I20260620 02:52:44.535513 6345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:44.617410 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 6222
I20260620 02:52:44.640825 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 6355
Jun 20 02:52:44 dist-test-slave-8p8c krb5kdc[5982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:52:44.657063 6506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:44.642169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57144 (local address 127.0.97.3:37369)
0620 02:52:44.642977 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:44.642984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:44.643018 (+ 34us) server_negotiation.cc:407] Connection header received
0620 02:52:44.643075 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:44.643082 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:44.643298 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:44.643433 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:44.644422 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:44.645469 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:44.646937 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:44.647498 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:44.647605 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:44.651752 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:44.651765 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:44.651780 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:44.651816 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:44.654635 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:44.655248 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:44.655255 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:44.655257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:44.655374 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:44.655725 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:44.655729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:44.655731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:44.656075 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:44.656318 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:44.656586 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 02:52:44.656780 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":469,"threads_started":1}
I20260620 02:52:44.658319 6478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:44.740183 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 6355
I20260620 02:52:44.755924 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 5995
I20260620 02:52:44.757144 6057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:44.861996 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 5995
2026-06-20T02:52:44Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3976 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:52:44 dist-test-slave-8p8c krb5kdc[6511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:52:44 dist-test-slave-8p8c krb5kdc[6511](info): set up 2 sockets
Jun 20 02:52:44 dist-test-slave-8p8c krb5kdc[6511](info): commencing operation
krb5kdc: starting...
W20260620 02:52:45.963240 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:52:46 dist-test-slave-8p8c krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:52:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:52:46Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260620 02:52:46.171269 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45057
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45057
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:46.494699 6524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:46.494989 6524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:46.495083 6524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:46.502648 6524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:52:46.502763 6524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:46.502810 6524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:46.502857 6524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:52:46.502907 6524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:52:46.513327 6524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:46.514701 6524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:46.516009 6524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:46.528646 6532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:46.528692 6530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:46.528606 6529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:46.529762 6524 server_base.cc:1034] running on GCE node
I20260620 02:52:46.530735 6524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:46.532354 6524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:46.533633 6524 hybrid_clock.cc:648] HybridClock initialized: now 1781923966533569 us; error 60 us; skew 500 ppm
I20260620 02:52:46.534013 6524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:46 dist-test-slave-8p8c krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923966, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:46.537652 6524 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260620 02:52:46.539323 6524 webserver.cc:460] Webserver started at http://127.0.97.62:34991/ using document root <none> and password file <none>
I20260620 02:52:46.539999 6524 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:46.540144 6524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:46.540441 6524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:46.543442 6524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "1caa9396c123467c8288bc9f054a1c1f"
format_stamp: "Formatted at 2026-06-20 02:52:46 on dist-test-slave-8p8c"
server_key: "9546b80c3f570ee0c1217b9db42ffd8f"
server_key_iv: "160f2ff868c08e8037fb0f659b1f87d9"
server_key_version: "encryptionkey@0"
I20260620 02:52:46.544173 6524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "1caa9396c123467c8288bc9f054a1c1f"
format_stamp: "Formatted at 2026-06-20 02:52:46 on dist-test-slave-8p8c"
server_key: "9546b80c3f570ee0c1217b9db42ffd8f"
server_key_iv: "160f2ff868c08e8037fb0f659b1f87d9"
server_key_version: "encryptionkey@0"
I20260620 02:52:46.549213 6524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:52:46.552682 6539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:46.553987 6524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:52:46.554198 6524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "1caa9396c123467c8288bc9f054a1c1f"
format_stamp: "Formatted at 2026-06-20 02:52:46 on dist-test-slave-8p8c"
server_key: "9546b80c3f570ee0c1217b9db42ffd8f"
server_key_iv: "160f2ff868c08e8037fb0f659b1f87d9"
server_key_version: "encryptionkey@0"
I20260620 02:52:46.554418 6524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:46.570847 6524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:46.575555 6524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:46.575855 6524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:46.595019 6524 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45057
I20260620 02:52:46.595103 6590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45057 every 8 connection(s)
I20260620 02:52:46.596463 6524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:52:46.600749 6591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:46.600751 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 6524
I20260620 02:52:46.600960 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:52:46.601617 388 external_mini_cluster.cc:1428] Setting key bf6c9226157d24caeb0b51b79e05d7a5
I20260620 02:52:46.616544 6591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f: Bootstrap starting.
Jun 20 02:52:46 dist-test-slave-8p8c krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:52:46.619990 6591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:46.621132 6591 log.cc:824] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:46.624282 6591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f: No bootstrap required, opened a new log
I20260620 02:52:46.625129 6594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:46.603522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60446 (local address 127.0.97.62:45057)
0620 02:52:46.604396 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:46.604407 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:46.604454 (+ 47us) server_negotiation.cc:407] Connection header received
0620 02:52:46.605933 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:46.605956 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:46.606356 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:46.606775 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:46.608834 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:46.610157 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:46.612182 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:46.612907 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:46.613057 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:46.617858 (+ 4801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:46.617866 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:46.617880 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:46.617923 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:46.620998 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:46.621723 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:46.621733 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:46.621739 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:46.621906 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:46.622352 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:46.622357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:46.622406 (+ 49us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:46.622940 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:46.623228 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:46.623668 (+ 440us) server_negotiation.cc:299] Negotiation successful
0620 02:52:46.624076 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":404,"threads_started":1}
I20260620 02:52:46.633100 6591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } }
I20260620 02:52:46.633504 6591 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:46.633628 6591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1caa9396c123467c8288bc9f054a1c1f, State: Initialized, Role: FOLLOWER
I20260620 02:52:46.634305 6591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } }
I20260620 02:52:46.634666 6591 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:52:46.634817 6591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:52:46.634984 6591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:46.637271 6591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } }
I20260620 02:52:46.637755 6591 leader_election.cc:302] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1caa9396c123467c8288bc9f054a1c1f; no voters:
I20260620 02:52:46.638351 6591 leader_election.cc:288] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:52:46.638540 6596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:46.640064 6596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [term 1 LEADER]: Becoming Leader. State: Replica: 1caa9396c123467c8288bc9f054a1c1f, State: Running, Role: LEADER
I20260620 02:52:46.640667 6596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } }
I20260620 02:52:46.640755 6591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:52:46.643882 6597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1caa9396c123467c8288bc9f054a1c1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } }
I20260620 02:52:46.643862 6598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1caa9396c123467c8288bc9f054a1c1f. Latest consensus state: current_term: 1 leader_uuid: "1caa9396c123467c8288bc9f054a1c1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } }
I20260620 02:52:46.644269 6597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:46.644302 6598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:46.646063 6605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:52:46.650563 6605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:52:46.659745 6605 catalog_manager.cc:1347] Generated new cluster ID: 13341c1efd294e0597c84690d18aca07
I20260620 02:52:46.659870 6605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:52:46.679239 6605 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:52:46.680626 6605 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:52:46.688687 6605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f: Generated new TSK 0
I20260620 02:52:46.689507 6605 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260620 02:52:46.738512 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45057
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:47.072311 6617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:47.072587 6617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:47.072741 6617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:47.079933 6617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:47.080045 6617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:47.080269 6617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:52:47.090179 6617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:47.091512 6617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:47.092837 6617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:47.105762 6622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:47.105885 6623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:47.105854 6625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:47.106642 6617 server_base.cc:1034] running on GCE node
I20260620 02:52:47.107223 6617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:47.108173 6617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:47.109395 6617 hybrid_clock.cc:648] HybridClock initialized: now 1781923967109325 us; error 58 us; skew 500 ppm
I20260620 02:52:47.109757 6617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:47 dist-test-slave-8p8c krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923967, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:47.112946 6617 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 02:52:47.114495 6617 webserver.cc:460] Webserver started at http://127.0.97.1:44557/ using document root <none> and password file <none>
I20260620 02:52:47.115180 6617 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:47.115322 6617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:47.115609 6617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:47.118462 6617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-0/data/instance:
uuid: "33858113a83d4f2da92c8f2364a93c83"
format_stamp: "Formatted at 2026-06-20 02:52:47 on dist-test-slave-8p8c"
server_key: "c5a1e83a08a8dc3a81241570b63fb197"
server_key_iv: "73118d98561e5369c037bd3c56a85948"
server_key_version: "encryptionkey@0"
I20260620 02:52:47.119227 6617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-0/wal/instance:
uuid: "33858113a83d4f2da92c8f2364a93c83"
format_stamp: "Formatted at 2026-06-20 02:52:47 on dist-test-slave-8p8c"
server_key: "c5a1e83a08a8dc3a81241570b63fb197"
server_key_iv: "73118d98561e5369c037bd3c56a85948"
server_key_version: "encryptionkey@0"
I20260620 02:52:47.124624 6617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:52:47.128373 6632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:47.129787 6617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:52:47.130090 6617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-0/wal
uuid: "33858113a83d4f2da92c8f2364a93c83"
format_stamp: "Formatted at 2026-06-20 02:52:47 on dist-test-slave-8p8c"
server_key: "c5a1e83a08a8dc3a81241570b63fb197"
server_key_iv: "73118d98561e5369c037bd3c56a85948"
server_key_version: "encryptionkey@0"
I20260620 02:52:47.130295 6617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:47.143478 6617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:47.148159 6617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:47.148456 6617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:47.149399 6617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:52:47.150751 6617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:47.150852 6617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:47.151001 6617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:47.151073 6617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:47.186580 6617 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43733
I20260620 02:52:47.186651 6744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43733 every 8 connection(s)
I20260620 02:52:47.188146 6617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:52:47.189277 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 6617
I20260620 02:52:47.189507 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:52:47.190313 388 external_mini_cluster.cc:1428] Setting key ef8bc2102282f610ab0e3f5a9c159bbd
Jun 20 02:52:47 dist-test-slave-8p8c krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923967, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:52:47.210497 6747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:47.191115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52027 (local address 127.0.97.62:45057)
0620 02:52:47.191536 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:47.191567 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:47.192634 (+ 1067us) server_negotiation.cc:407] Connection header received
0620 02:52:47.194438 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:47.194443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:47.194529 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:47.194649 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:47.196143 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:47.196827 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:47.198705 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:47.199257 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:47.199411 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:47.202871 (+ 3460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:47.202881 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:47.202890 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:47.202938 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:47.206940 (+ 4002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:47.207689 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:47.207696 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:47.207701 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:47.207866 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:47.208357 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:47.208363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:47.208368 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:47.208823 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:47.209026 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:47.209732 (+ 706us) server_negotiation.cc:299] Negotiation successful
0620 02:52:47.210012 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":222,"threads_started":1}
I20260620 02:52:47.211306 6748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:47.191666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.97.1:52027)
0620 02:52:47.192473 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:47.192517 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:47.194162 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:47.194882 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:47.194891 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:47.195402 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:47.195964 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:47.195981 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:47.196986 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:47.196991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:47.198547 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:47.198563 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:47.199459 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:47.199464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:47.199671 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:47.200401 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:47.200423 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:47.202640 (+ 2217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:47.207093 (+ 4453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:47.207100 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:47.207113 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:47.207556 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:47.208011 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:47.208014 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:47.208017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:47.208233 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:47.208990 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:47.208996 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:47.209318 (+ 322us) client_negotiation.cc:769] Sending connection context
0620 02:52:47.209573 (+ 255us) client_negotiation.cc:240] Negotiation successful
0620 02:52:47.210121 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":361,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:52:47.212864 6745 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45057
I20260620 02:52:47.213201 6745 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:47.213964 6745 heartbeater.cc:511] Master 127.0.97.62:45057 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
I20260620 02:52:47.216480 6556 ts_manager.cc:194] Registered new tserver with Master: 33858113a83d4f2da92c8f2364a93c83 (127.0.97.1:43733)
I20260620 02:52:47.218603 6556 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260620 02:52:47.241681 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45057
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:47.569607 6751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:47.569933 6751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:47.570112 6751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:47.577332 6751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:47.577487 6751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:47.577678 6751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:52:47.587488 6751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:47.588846 6751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:47.590179 6751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:47.604179 6756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:47.604255 6757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:47.604347 6759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:47.605283 6751 server_base.cc:1034] running on GCE node
I20260620 02:52:47.605959 6751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:47.607024 6751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:47.608278 6751 hybrid_clock.cc:648] HybridClock initialized: now 1781923967608219 us; error 49 us; skew 500 ppm
I20260620 02:52:47.608593 6751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:47 dist-test-slave-8p8c krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923967, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:47.612000 6751 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 02:52:47.613461 6751 webserver.cc:460] Webserver started at http://127.0.97.2:32999/ using document root <none> and password file <none>
I20260620 02:52:47.614137 6751 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:47.614279 6751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:47.614708 6751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:47.617466 6751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-1/data/instance:
uuid: "66f25b3d96fa40f294be341e1266df2d"
format_stamp: "Formatted at 2026-06-20 02:52:47 on dist-test-slave-8p8c"
server_key: "6de2c406956d0933721b3fb94536d981"
server_key_iv: "bf1500b6d9704b6f897c39f33fb9ca0c"
server_key_version: "encryptionkey@0"
I20260620 02:52:47.618219 6751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-1/wal/instance:
uuid: "66f25b3d96fa40f294be341e1266df2d"
format_stamp: "Formatted at 2026-06-20 02:52:47 on dist-test-slave-8p8c"
server_key: "6de2c406956d0933721b3fb94536d981"
server_key_iv: "bf1500b6d9704b6f897c39f33fb9ca0c"
server_key_version: "encryptionkey@0"
I20260620 02:52:47.623109 6751 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:52:47.626495 6766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:47.627934 6751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:52:47.628160 6751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-1/wal
uuid: "66f25b3d96fa40f294be341e1266df2d"
format_stamp: "Formatted at 2026-06-20 02:52:47 on dist-test-slave-8p8c"
server_key: "6de2c406956d0933721b3fb94536d981"
server_key_iv: "bf1500b6d9704b6f897c39f33fb9ca0c"
server_key_version: "encryptionkey@0"
I20260620 02:52:47.628409 6751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:47.650329 6751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:47.655411 6751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:47.655761 6751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:47.656873 6751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:52:47.659400 6751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:47.659519 6751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:47.659677 6751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:47.659754 6751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:47.713778 6751 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36091
I20260620 02:52:47.713869 6878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36091 every 8 connection(s)
I20260620 02:52:47.715210 6751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:52:47.723559 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 6751
I20260620 02:52:47.723742 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:52:47.724395 388 external_mini_cluster.cc:1428] Setting key 47c8ee2cbf472319583115936f1cf3ab
Jun 20 02:52:47 dist-test-slave-8p8c krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923967, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:52:47.740944 6881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:47.718546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41105 (local address 127.0.97.62:45057)
0620 02:52:47.719181 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:47.719190 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:47.719903 (+ 713us) server_negotiation.cc:407] Connection header received
0620 02:52:47.721732 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:47.721739 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:47.721833 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:47.721977 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:47.724112 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:47.725075 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:47.727777 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:47.728540 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:47.728684 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:47.733535 (+ 4851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:47.733541 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:47.733544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:47.733577 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:47.737317 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:47.738198 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:47.738207 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:47.738213 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:47.738404 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:47.738852 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:47.738859 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:47.738865 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:47.739267 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:47.739421 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:47.740193 (+ 772us) server_negotiation.cc:299] Negotiation successful
0620 02:52:47.740448 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":347,"threads_started":1}
I20260620 02:52:47.741783 6882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:47.718891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.97.2:41105)
0620 02:52:47.719735 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:47.719786 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:47.721460 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:47.722266 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:47.722276 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:47.722912 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:47.723905 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:47.723927 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:47.725257 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:47.725264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:47.727600 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:47.727617 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:47.728803 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:47.728812 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:47.729150 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:47.730005 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:47.730043 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:47.733227 (+ 3184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:47.737523 (+ 4296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:47.737531 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:47.737545 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:47.738039 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:47.738523 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:47.738527 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:47.738529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:47.738731 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:47.739414 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:47.739420 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:47.739774 (+ 354us) client_negotiation.cc:769] Sending connection context
0620 02:52:47.740048 (+ 274us) client_negotiation.cc:240] Negotiation successful
0620 02:52:47.740477 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":344,"threads_started":1}
I20260620 02:52:47.743492 6879 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45057
I20260620 02:52:47.744056 6879 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:47.745148 6879 heartbeater.cc:511] Master 127.0.97.62:45057 requested a full tablet report, sending...
I20260620 02:52:47.746832 6556 ts_manager.cc:194] Registered new tserver with Master: 66f25b3d96fa40f294be341e1266df2d (127.0.97.2:36091)
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:52:47.747951 6556 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41105
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260620 02:52:47.782770 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45057
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:48.115599 6885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:48.115926 6885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:48.116096 6885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:48.123503 6885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:48.123651 6885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:48.123818 6885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:52:48.133837 6885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:48.135260 6885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:48.136567 6885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:48.150231 6890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:48.150329 6893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:48.150249 6891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:48.151111 6885 server_base.cc:1034] running on GCE node
I20260620 02:52:48.151736 6885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:48.152772 6885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:48.154035 6885 hybrid_clock.cc:648] HybridClock initialized: now 1781923968153980 us; error 44 us; skew 500 ppm
I20260620 02:52:48.154459 6885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:48 dist-test-slave-8p8c krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923968, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:48.157588 6885 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 02:52:48.159482 6885 webserver.cc:460] Webserver started at http://127.0.97.3:35109/ using document root <none> and password file <none>
I20260620 02:52:48.160140 6885 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:48.160239 6885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:48.160485 6885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:48.163453 6885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-2/data/instance:
uuid: "56dec7a4e0b8482aa22edc244ee37c51"
format_stamp: "Formatted at 2026-06-20 02:52:48 on dist-test-slave-8p8c"
server_key: "49f44d5943586e36e36e5c05610a9a74"
server_key_iv: "b9d350a1cc57643abb4e1d7548e0c355"
server_key_version: "encryptionkey@0"
I20260620 02:52:48.164177 6885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-2/wal/instance:
uuid: "56dec7a4e0b8482aa22edc244ee37c51"
format_stamp: "Formatted at 2026-06-20 02:52:48 on dist-test-slave-8p8c"
server_key: "49f44d5943586e36e36e5c05610a9a74"
server_key_iv: "b9d350a1cc57643abb4e1d7548e0c355"
server_key_version: "encryptionkey@0"
I20260620 02:52:48.169524 6885 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:52:48.173493 6900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:48.175463 6885 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260620 02:52:48.175652 6885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-2/wal
uuid: "56dec7a4e0b8482aa22edc244ee37c51"
format_stamp: "Formatted at 2026-06-20 02:52:48 on dist-test-slave-8p8c"
server_key: "49f44d5943586e36e36e5c05610a9a74"
server_key_iv: "b9d350a1cc57643abb4e1d7548e0c355"
server_key_version: "encryptionkey@0"
I20260620 02:52:48.175865 6885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:48.191808 6885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:48.196247 6885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:48.196540 6885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:48.197523 6885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:52:48.198946 6885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:48.199024 6885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:48.199148 6885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:48.199196 6885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:48.222576 6745 heartbeater.cc:503] Master 127.0.97.62:45057 was elected leader, sending a full tablet report...
I20260620 02:52:48.235052 6885 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43517
I20260620 02:52:48.235147 7012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43517 every 8 connection(s)
I20260620 02:52:48.236342 6885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:52:48.243932 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 6885
I20260620 02:52:48.244113 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:52:48.244668 388 external_mini_cluster.cc:1428] Setting key 63de67736972441cc944762f4b20b05e
Jun 20 02:52:48 dist-test-slave-8p8c krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923968, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:52:48.257539 6881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:48.239140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53253 (local address 127.0.97.62:45057)
0620 02:52:48.239334 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:48.239339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:48.240552 (+ 1213us) server_negotiation.cc:407] Connection header received
0620 02:52:48.242583 (+ 2031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:48.242588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:48.242676 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:48.242768 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:48.244523 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:48.245630 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:48.247823 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:48.248218 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:48.248351 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:48.251806 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:48.251811 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:48.251814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:48.251845 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:48.254578 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:48.255347 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:48.255352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:48.255354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:48.255485 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:48.255894 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:48.255925 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:48.255927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:48.256207 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:48.256321 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:48.257050 (+ 729us) server_negotiation.cc:299] Negotiation successful
0620 02:52:48.257283 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 02:52:48.258558 7015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:48.239641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.97.3:53253)
0620 02:52:48.240345 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:48.240431 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:48.242235 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:48.242971 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:48.242979 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:48.243457 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:48.244335 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:48.244362 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:48.245783 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:48.245788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:48.247683 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:48.247695 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:48.248435 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:48.248442 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:48.248671 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:48.249355 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:48.249377 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:48.251603 (+ 2226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:48.254726 (+ 3123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:48.254738 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:48.254753 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:48.255209 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:48.255594 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:48.255598 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:48.255600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:48.255779 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:48.256328 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:48.256334 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:48.256646 (+ 312us) client_negotiation.cc:769] Sending connection context
0620 02:52:48.256867 (+ 221us) client_negotiation.cc:240] Negotiation successful
0620 02:52:48.257349 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":283,"threads_started":1}
I20260620 02:52:48.260026 7013 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45057
I20260620 02:52:48.260433 7013 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:48.261142 7013 heartbeater.cc:511] Master 127.0.97.62:45057 requested a full tablet report, sending...
I20260620 02:52:48.262523 6556 ts_manager.cc:194] Registered new tserver with Master: 56dec7a4e0b8482aa22edc244ee37c51 (127.0.97.3:43517)
I20260620 02:52:48.263289 6556 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53253
I20260620 02:52:48.272786 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260620 02:52:48.322868 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:41347
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45057,127.0.97.61:41347
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:48.669351 7018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:48.669741 7018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:48.669921 7018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:48.677979 7018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:52:48.678140 7018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:48.678252 7018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:48.678344 7018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:52:48.678455 7018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:52:48.688665 7018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45057,127.0.97.61:41347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:41347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
W20260620 02:52:48.689095 7018 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:45057,127.0.97.61:41347'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 02:52:48.690186 7018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:48.691668 7018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:48.703526 7024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:48.703918 7023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:48.703989 7026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:48.707962 7018 server_base.cc:1034] running on GCE node
I20260620 02:52:48.708737 7018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:48.710162 7018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:48.711413 7018 hybrid_clock.cc:648] HybridClock initialized: now 1781923968711323 us; error 56 us; skew 500 ppm
I20260620 02:52:48.711829 7018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:48 dist-test-slave-8p8c krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923968, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:48.720722 7018 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260620 02:52:48.722235 7018 webserver.cc:460] Webserver started at http://127.0.97.61:33149/ using document root <none> and password file <none>
I20260620 02:52:48.723011 7018 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:48.723117 7018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:48.723379 7018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:48.726300 7018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-1/data/instance:
uuid: "982c8b2c204d443087f6c9f667437974"
format_stamp: "Formatted at 2026-06-20 02:52:48 on dist-test-slave-8p8c"
server_key: "ca7a28aa41e6d0dee53a9dc58f1c97c4"
server_key_iv: "ff3127fb54569af19e8befcedf97c62e"
server_key_version: "encryptionkey@0"
I20260620 02:52:48.727192 7018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-1/wal/instance:
uuid: "982c8b2c204d443087f6c9f667437974"
format_stamp: "Formatted at 2026-06-20 02:52:48 on dist-test-slave-8p8c"
server_key: "ca7a28aa41e6d0dee53a9dc58f1c97c4"
server_key_iv: "ff3127fb54569af19e8befcedf97c62e"
server_key_version: "encryptionkey@0"
I20260620 02:52:48.732312 7018 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:52:48.735906 7033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:48.737381 7018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:52:48.737571 7018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-1/wal
uuid: "982c8b2c204d443087f6c9f667437974"
format_stamp: "Formatted at 2026-06-20 02:52:48 on dist-test-slave-8p8c"
server_key: "ca7a28aa41e6d0dee53a9dc58f1c97c4"
server_key_iv: "ff3127fb54569af19e8befcedf97c62e"
server_key_version: "encryptionkey@0"
I20260620 02:52:48.737766 7018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:48.751302 6879 heartbeater.cc:503] Master 127.0.97.62:45057 was elected leader, sending a full tablet report...
I20260620 02:52:48.760229 7018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:48.765017 7018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:48.765328 7018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:48.786054 7018 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:41347
I20260620 02:52:48.786096 7084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:41347 every 8 connection(s)
I20260620 02:52:48.787504 7018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-1/data/info.pb
I20260620 02:52:48.791640 7085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:48.795403 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 7018
I20260620 02:52:48.800016 7085 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:52:48 dist-test-slave-8p8c krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 20 02:52:48 dist-test-slave-8p8c krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923968, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:52:48.821086 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:48.804614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60452 (local address 127.0.97.62:45057)
0620 02:52:48.805814 (+ 1200us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:48.805820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:48.805839 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:52:48.805899 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:48.805904 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:48.805989 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:48.806102 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:48.807540 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:48.808745 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:48.810167 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:48.810874 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:48.811022 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:48.815222 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:48.815231 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:48.815237 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:48.815284 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:48.817879 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:48.818691 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:48.818698 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:48.818701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:48.818840 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:48.819338 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:48.819343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:48.819346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:48.819699 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:48.819807 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:48.820520 (+ 713us) server_negotiation.cc:299] Negotiation successful
0620 02:52:48.820686 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1069,"thread_start_us":947,"threads_started":1}
I20260620 02:52:48.821589 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:48.797877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59820 (local address 127.0.97.61:41347)
0620 02:52:48.798801 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:48.798814 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:48.798854 (+ 40us) server_negotiation.cc:407] Connection header received
0620 02:52:48.800668 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:48.800691 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:48.801134 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:48.801534 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:48.803320 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:48.805429 (+ 2109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:48.807098 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:48.807935 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:48.808072 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:48.813222 (+ 5150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:48.813236 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:48.813260 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:48.813321 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:48.816510 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:48.817192 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:48.817199 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:48.817206 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:48.817354 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:48.817747 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:48.817755 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:48.817757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:48.818483 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:48.818761 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:48.819367 (+ 606us) server_negotiation.cc:299] Negotiation successful
0620 02:52:48.819824 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":494,"threads_started":1}
I20260620 02:52:48.821611 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:48.804209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.0.1:60452)
0620 02:52:48.804925 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:48.804982 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:48.805129 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:48.806253 (+ 1124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:48.806258 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:48.806951 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:48.807342 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:48.807352 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:48.808920 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:48.808926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:48.810014 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:48.810025 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:48.811164 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:48.811170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:48.811307 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:48.812046 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:48.812080 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:48.814982 (+ 2902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:48.817998 (+ 3016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:48.818006 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:48.818013 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:48.818586 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:48.819007 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:48.819013 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:48.819017 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:48.819203 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:48.819820 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:48.819827 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:48.819989 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 02:52:48.820351 (+ 362us) client_negotiation.cc:240] Negotiation successful
0620 02:52:48.820754 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":564,"threads_started":1}
I20260620 02:52:48.824882 7085 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:52:48 dist-test-slave-8p8c krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923968, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 02:52:48.838546 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:48.825625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41347 (local address 127.0.0.1:59830)
0620 02:52:48.825800 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:48.825820 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:48.825973 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:48.826285 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:48.826289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:48.826705 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:48.826836 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:48.826844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:48.827862 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:48.827866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:48.828918 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:48.828930 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:48.829702 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:48.829708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:48.829871 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:48.830558 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:48.830580 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:48.832748 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:48.836241 (+ 3493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:48.836245 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:48.836247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:48.836630 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:48.837034 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:48.837038 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:48.837041 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:48.837184 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:48.837737 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:48.837743 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:48.837874 (+ 131us) client_negotiation.cc:769] Sending connection context
0620 02:52:48.838004 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 02:52:48.838214 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 02:52:48.838660 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:48.825844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59830 (local address 127.0.97.61:41347)
0620 02:52:48.826015 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:48.826021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:48.826041 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:52:48.826082 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:48.826085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:48.826141 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:48.826225 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:48.826979 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:48.827722 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:48.829108 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:48.829528 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:48.829642 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:48.832999 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:48.833007 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:48.833012 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:48.833059 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:48.836105 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:48.836746 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:48.836750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:48.836753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:48.836868 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:48.837312 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:48.837316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:48.837318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:48.837608 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:48.837750 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:48.838160 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 02:52:48.838316 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 02:52:48.843698 7085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974: Bootstrap starting.
I20260620 02:52:48.846931 7085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:48.847955 7085 log.cc:824] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:48.850859 7085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974: No bootstrap required, opened a new log
I20260620 02:52:48.857789 7085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } }
I20260620 02:52:48.858148 7085 raft_consensus.cc:374] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:48.858244 7085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982c8b2c204d443087f6c9f667437974, State: Initialized, Role: FOLLOWER
I20260620 02:52:48.858903 7085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } }
I20260620 02:52:48.859853 7092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } } }
I20260620 02:52:48.860231 7092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:52:48.860852 7085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 02:52:48.870532 7103 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:52:48.870769 7103 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:52:48.872709 7018 master_runner.cc:428] Detected that this master 982c8b2c204d443087f6c9f667437974 is joining an existing cluster
I20260620 02:52:48.872835 7018 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:41347
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260620 02:52:48.903739 7109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:48.877157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.0.1:60466)
0620 02:52:48.884239 (+ 7082us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:48.884256 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:48.884458 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:48.885606 (+ 1148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:48.885610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:48.886498 (+ 888us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:48.886963 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:48.886976 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:48.888628 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:48.888637 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:48.890888 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:48.890904 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:48.892266 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:48.892275 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:48.892528 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:48.893429 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:48.893466 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:48.894998 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:48.900494 (+ 5496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:48.900501 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:48.900510 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:48.901033 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:48.901506 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:48.901510 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:48.901514 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:48.901686 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:48.902396 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:48.902403 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:48.902753 (+ 350us) client_negotiation.cc:769] Sending connection context
0620 02:52:48.902925 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 02:52:48.903233 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6927,"mutex_wait_us":284,"spinlock_wait_cycles":37632,"thread_start_us":6767,"threads_started":1}
I20260620 02:52:48.903743 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:48.877284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60466 (local address 127.0.97.62:45057)
0620 02:52:48.877489 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:48.877497 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:48.884368 (+ 6871us) server_negotiation.cc:407] Connection header received
0620 02:52:48.884655 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:48.884664 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:48.884785 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:48.884881 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:48.887170 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:48.888439 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:48.891072 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:48.891813 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:48.891933 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:48.895215 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:48.895221 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:48.895226 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:48.895267 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:48.900320 (+ 5053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:48.901195 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:48.901203 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:48.901208 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:48.901377 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:48.901805 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:48.901813 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:48.901820 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:48.902237 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:48.902419 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:48.903122 (+ 703us) server_negotiation.cc:299] Negotiation successful
0620 02:52:48.903382 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:52:48.904230 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:48.884594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59836 (local address 127.0.97.61:41347)
0620 02:52:48.885472 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:48.885478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:48.885538 (+ 60us) server_negotiation.cc:407] Connection header received
0620 02:52:48.885747 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:48.885753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:48.885832 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:48.885925 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:48.887471 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:48.888617 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:48.891064 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:48.891860 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:48.891948 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:48.895184 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:48.895193 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:48.895198 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:48.895237 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:48.900484 (+ 5247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:48.901570 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:48.901576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:48.901579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:48.901752 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:48.902177 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:48.902182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:48.902186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:48.902631 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:48.902776 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:48.903595 (+ 819us) server_negotiation.cc:299] Negotiation successful
0620 02:52:48.903844 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":451}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
I20260620 02:52:48.905225 7111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:48.884566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41347 (local address 127.0.0.1:59836)
0620 02:52:48.885455 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:48.885475 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:48.885613 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:48.886172 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:48.886176 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:48.886963 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:48.887331 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:48.887343 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:48.888857 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:48.888863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:48.890888 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:48.890904 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:48.892287 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:48.892293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:48.892533 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:48.893562 (+ 1029us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:48.893588 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:48.894998 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:48.900780 (+ 5782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:48.900785 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:48.900790 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:48.901419 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:48.901885 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:48.901893 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:48.901899 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:48.902051 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:48.902830 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:48.902833 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:48.902993 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 02:52:48.904393 (+ 1400us) client_negotiation.cc:240] Negotiation successful
0620 02:52:48.904720 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":222,"thread_start_us":277,"threads_started":1}
Jun 20 02:52:48 dist-test-slave-8p8c krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923966, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 02:52:48.928223 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:48.911628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59848 (local address 127.0.97.61:41347)
0620 02:52:48.911961 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:48.911969 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:48.912424 (+ 455us) server_negotiation.cc:407] Connection header received
0620 02:52:48.912917 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:48.912924 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:48.913042 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:48.913169 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:48.914717 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:48.915491 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:48.916933 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:48.917452 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:48.917623 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:48.921509 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:48.921516 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:48.921519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:48.921556 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:48.924871 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:48.925707 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:48.925716 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:48.925726 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:48.925878 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:48.926333 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:48.926341 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:48.926347 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:48.926732 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:48.926915 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:48.927479 (+ 564us) server_negotiation.cc:299] Negotiation successful
0620 02:52:48.927750 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260620 02:52:48.928285 7112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:48.911558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41347 (local address 127.0.0.1:59848)
0620 02:52:48.912286 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:48.912329 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:48.912570 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:48.913343 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:48.913352 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:48.914215 (+ 863us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:48.914550 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:48.914563 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:48.915651 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:48.915657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:48.916779 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:48.916791 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:48.917644 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:48.917651 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:48.917857 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:48.918660 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:48.918683 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:48.921271 (+ 2588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:48.925046 (+ 3775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:48.925061 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:48.925070 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:48.925553 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:48.926017 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:48.926025 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:48.926030 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:48.926199 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:48.926914 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:48.926925 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:48.927144 (+ 219us) client_negotiation.cc:769] Sending connection context
0620 02:52:48.927369 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 02:52:48.927762 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":407,"threads_started":1}
I20260620 02:52:48.930791 6556 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:41347: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "982c8b2c204d443087f6c9f667437974"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 41347
}
attrs {
promote: true
}
}
dest_uuid: "1caa9396c123467c8288bc9f054a1c1f"
cas_config_opid_index: -1
I20260620 02:52:48.931969 6556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: true } }
I20260620 02:52:48.936733 7116 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 982c8b2c204d443087f6c9f667437974 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: true } } }
I20260620 02:52:48.938109 7060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:48.941543 7116 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:41347
I20260620 02:52:48.941998 7060 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [term 1 FOLLOWER]: Refusing update from remote peer 1caa9396c123467c8288bc9f054a1c1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 02:52:48.942340 7018 master.cc:537] Master@127.0.97.61:41347 shutting down...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260620 02:52:48.947778 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:43075
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45057,127.0.97.61:41347,127.0.97.60:43075
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:52:48.948942 7115 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:52:48.949079 7117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1caa9396c123467c8288bc9f054a1c1f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: true } } }
I20260620 02:52:48.949688 7117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [sys.catalog]: This master's current role is: LEADER
W20260620 02:52:48.955185 6541 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260620 02:52:48.957322 7018 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:52:48.957888 7018 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 02:52:48.957970 6541 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f -> Peer 982c8b2c204d443087f6c9f667437974 (127.0.97.61:41347): Couldn't send request to peer 982c8b2c204d443087f6c9f667437974. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 02:52:48.958240 7018 tablet_replica.cc:330] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974: stopping tablet replica
I20260620 02:52:48.965134 7018 master.cc:552] Master@127.0.97.61:41347 shutdown complete.
I20260620 02:52:48.965296 7018 master_runner.cc:305] Clearing existing system tablet
I20260620 02:52:48.965801 7018 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:48.965972 7018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:48.969295 7018 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 02:52:48.971302 7122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:48.971710 7018 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:48.972151 7018 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260620 02:52:48.972286 7018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-1/wal
uuid: "982c8b2c204d443087f6c9f667437974"
format_stamp: "Formatted at 2026-06-20 02:52:48 on dist-test-slave-8p8c"
server_key: "ca7a28aa41e6d0dee53a9dc58f1c97c4"
server_key_iv: "ff3127fb54569af19e8befcedf97c62e"
server_key_version: "encryptionkey@0"
I20260620 02:52:48.978430 7018 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:48.986758 7018 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 02:52:48.987100 7018 log.cc:1196] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 02:52:48.987669 7018 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974: Deleting consensus metadata
I20260620 02:52:48.987989 7018 master_runner.cc:315] Copying system tablet from 127.0.97.62:45057
I20260620 02:52:48.990736 7018 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:45057
I20260620 02:52:49.015588 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:48.991677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60472 (local address 127.0.97.62:45057)
0620 02:52:48.991883 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:48.991889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:49.000598 (+ 8709us) server_negotiation.cc:407] Connection header received
0620 02:52:49.000777 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:49.000783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:49.000888 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:49.001008 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:49.002148 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.003339 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.005321 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.005989 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.006124 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:49.008524 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:49.008532 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:49.008539 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:49.008581 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:49.012054 (+ 3473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:49.013070 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:49.013075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:49.013079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:49.013242 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:49.013787 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:49.013793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:49.013796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:49.014169 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:49.014323 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:49.014911 (+ 588us) server_negotiation.cc:299] Negotiation successful
0620 02:52:49.015155 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 02:52:49.015816 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:48.991597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.0.1:60472)
0620 02:52:49.000403 (+ 8806us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:49.000436 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:49.000637 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:49.001065 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:49.001070 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:49.001642 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:49.001970 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:49.001984 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.003610 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.003616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:49.005146 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:49.005161 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.006157 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.006163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:49.006342 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:49.007157 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:49.007186 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:49.008287 (+ 1101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:49.012279 (+ 3992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:49.012284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:49.012288 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:49.012793 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:49.013398 (+ 605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:49.013431 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:49.013437 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:49.013654 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:49.014320 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:49.014326 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:49.014611 (+ 285us) client_negotiation.cc:769] Sending connection context
0620 02:52:49.014805 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 02:52:49.015255 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8654,"thread_start_us":8137,"threads_started":1}
I20260620 02:52:49.016986 6576 tablet_copy_service.cc:140] P 1caa9396c123467c8288bc9f054a1c1f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 982c8b2c204d443087f6c9f667437974 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:60472)
I20260620 02:52:49.017648 6576 tablet_copy_service.cc:161] P 1caa9396c123467c8288bc9f054a1c1f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 982c8b2c204d443087f6c9f667437974 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:60472: session id = 982c8b2c204d443087f6c9f667437974-00000000000000000000000000000000
I20260620 02:52:49.022425 6576 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 02:52:49.024852 7018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 02:52:49.032204 7018 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974: tablet copy: Starting download of 0 data blocks...
I20260620 02:52:49.032546 7018 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974: tablet copy: Starting download of 1 WAL segments...
I20260620 02:52:49.035395 7018 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 02:52:49.040530 6576 tablet_copy_service.cc:342] P 1caa9396c123467c8288bc9f054a1c1f: Request end of tablet copy session 982c8b2c204d443087f6c9f667437974-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:60472
I20260620 02:52:49.040766 6576 tablet_copy_service.cc:433] P 1caa9396c123467c8288bc9f054a1c1f: ending tablet copy session 982c8b2c204d443087f6c9f667437974-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 982c8b2c204d443087f6c9f667437974
I20260620 02:52:49.053227 7018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:49.056891 7129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:49.062785 7130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:49.067137 7132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:49.069830 7018 server_base.cc:1034] running on GCE node
I20260620 02:52:49.070204 7018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:49.072924 7018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:49.074157 7018 hybrid_clock.cc:648] HybridClock initialized: now 1781923969074048 us; error 77 us; skew 500 ppm
I20260620 02:52:49.074458 7018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:49 dist-test-slave-8p8c krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923969, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:49.076763 7018 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260620 02:52:49.078321 7018 webserver.cc:460] Webserver started at http://127.0.97.61:36979/ using document root <none> and password file <none>
I20260620 02:52:49.078886 7018 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:49.078979 7018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:49.082444 7018 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20260620 02:52:49.084559 7139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:49.084992 7018 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 02:52:49.085177 7018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-1/wal
uuid: "982c8b2c204d443087f6c9f667437974"
format_stamp: "Formatted at 2026-06-20 02:52:48 on dist-test-slave-8p8c"
server_key: "ca7a28aa41e6d0dee53a9dc58f1c97c4"
server_key_iv: "ff3127fb54569af19e8befcedf97c62e"
server_key_version: "encryptionkey@0"
I20260620 02:52:49.085413 7018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:49.112509 7018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:49.117167 7018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:49.138044 7018 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:41347
I20260620 02:52:49.138134 7190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:41347 every 8 connection(s)
I20260620 02:52:49.139158 7018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-1/data/info.pb
I20260620 02:52:49.144851 7191 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:52:49.150171 7191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974: Bootstrap starting.
I20260620 02:52:49.170509 7191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:52:49.171078 7191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974: Bootstrap complete.
I20260620 02:52:49.172144 7191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: true } }
I20260620 02:52:49.172569 7191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 982c8b2c204d443087f6c9f667437974, State: Initialized, Role: LEARNER
I20260620 02:52:49.172860 7191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: true } }
I20260620 02:52:49.174149 7191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:52:49.174006 7194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: true } } }
I20260620 02:52:49.174553 7194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [sys.catalog]: This master's current role is: LEARNER
I20260620 02:52:49.186941 7205 catalog_manager.cc:1259] Loaded cluster ID: 13341c1efd294e0597c84690d18aca07
I20260620 02:52:49.187129 7205 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974: loading cluster ID for follower catalog manager: success
I20260620 02:52:49.190858 7205 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974: acquiring CA information for follower catalog manager: success
I20260620 02:52:49.193161 7205 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:52:49.265748 7013 heartbeater.cc:503] Master 127.0.97.62:45057 was elected leader, sending a full tablet report...
W20260620 02:52:49.351078 7120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:49.351364 7120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:49.351503 7120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:49.359480 7120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:52:49.359589 7120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:49.359664 7120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:49.359761 7120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:52:49.359820 7120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:52:49.370277 7120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45057,127.0.97.61:41347,127.0.97.60:43075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:43075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:49.371704 7120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:49.373086 7120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:49.386658 7213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:49.386711 7211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:49.387060 7210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:49.392143 7120 server_base.cc:1034] running on GCE node
I20260620 02:52:49.393006 7120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:49.394609 7120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:49.395809 7120 hybrid_clock.cc:648] HybridClock initialized: now 1781923969395766 us; error 38 us; skew 500 ppm
I20260620 02:52:49.396162 7120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:49 dist-test-slave-8p8c krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923969, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:49.399412 7120 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260620 02:52:49.401328 7120 webserver.cc:460] Webserver started at http://127.0.97.60:46249/ using document root <none> and password file <none>
I20260620 02:52:49.402042 7120 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:49.402179 7120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:49.402541 7120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:49.405387 7120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-2/data/instance:
uuid: "8d11130cfec9434094b0340a81f4230e"
format_stamp: "Formatted at 2026-06-20 02:52:49 on dist-test-slave-8p8c"
server_key: "27fb06d27785e57cc7edf3094ea49e38"
server_key_iv: "4827dd067c24c2ff6edd27f56adfdb86"
server_key_version: "encryptionkey@0"
I20260620 02:52:49.406136 7120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-2/wal/instance:
uuid: "8d11130cfec9434094b0340a81f4230e"
format_stamp: "Formatted at 2026-06-20 02:52:49 on dist-test-slave-8p8c"
server_key: "27fb06d27785e57cc7edf3094ea49e38"
server_key_iv: "4827dd067c24c2ff6edd27f56adfdb86"
server_key_version: "encryptionkey@0"
I20260620 02:52:49.411305 7120 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 02:52:49.415127 7220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:49.416571 7120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:52:49.416745 7120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-2/wal
uuid: "8d11130cfec9434094b0340a81f4230e"
format_stamp: "Formatted at 2026-06-20 02:52:49 on dist-test-slave-8p8c"
server_key: "27fb06d27785e57cc7edf3094ea49e38"
server_key_iv: "4827dd067c24c2ff6edd27f56adfdb86"
server_key_version: "encryptionkey@0"
I20260620 02:52:49.416934 7120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:49.438639 7120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:49.444098 7120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:49.444393 7120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:49.465173 7120 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:43075
I20260620 02:52:49.465209 7271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:43075 every 8 connection(s)
I20260620 02:52:49.466739 7120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-2/data/info.pb
I20260620 02:52:49.468616 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 7120
I20260620 02:52:49.471686 7272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:49.480532 7272 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:52:49 dist-test-slave-8p8c krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 20 02:52:49 dist-test-slave-8p8c krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923969, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:52:49.498993 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:49.471214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54702 (local address 127.0.97.60:43075)
0620 02:52:49.471963 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:49.471979 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:49.472034 (+ 55us) server_negotiation.cc:407] Connection header received
0620 02:52:49.474251 (+ 2217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:49.474278 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:49.474812 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:49.475268 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:49.477142 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.479254 (+ 2112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.480640 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.481707 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.481884 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:49.487413 (+ 5529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:49.487423 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:49.487436 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:49.487494 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:49.492295 (+ 4801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:49.493399 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:49.493412 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:49.493425 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:49.493674 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:49.494230 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:49.494237 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:49.494243 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:49.495308 (+ 1065us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:49.495646 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:49.496560 (+ 914us) server_negotiation.cc:299] Negotiation successful
0620 02:52:49.497146 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":297,"threads_started":1}
I20260620 02:52:49.504683 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:49.483627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60484 (local address 127.0.97.62:45057)
0620 02:52:49.483823 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:49.483831 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:49.484766 (+ 935us) server_negotiation.cc:407] Connection header received
0620 02:52:49.484987 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:49.484991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:49.485054 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:49.485157 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:49.486507 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.487385 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.489712 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.490647 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.490776 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:49.496356 (+ 5580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:49.496366 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:49.496374 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:49.496430 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:49.501003 (+ 4573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:49.502297 (+ 1294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:49.502305 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:49.502312 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:49.502536 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:49.502991 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:49.502995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:49.502999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:49.503346 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:49.503458 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:49.504084 (+ 626us) server_negotiation.cc:299] Negotiation successful
0620 02:52:49.504297 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 02:52:49.505071 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:49.483499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.0.1:60484)
0620 02:52:49.484236 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:49.484278 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:49.484848 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:49.485211 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:49.485218 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:49.486011 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:49.486354 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:49.486389 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.487530 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.487536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:49.489506 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:49.489525 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.491160 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.491170 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:49.491441 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:49.492331 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:49.492363 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:49.496088 (+ 3725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:49.501253 (+ 5165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:49.501269 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:49.501281 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:49.502162 (+ 881us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:49.502654 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:49.502659 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:49.502662 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:49.502842 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:49.503567 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:49.503575 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:49.503743 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 02:52:49.503963 (+ 220us) client_negotiation.cc:240] Negotiation successful
0620 02:52:49.504319 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":345,"threads_started":1}
I20260620 02:52:49.506512 7272 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:52:49.508931 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:49.488573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41347 (local address 127.0.0.1:59864)
0620 02:52:49.502027 (+ 13454us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:49.502047 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:49.502236 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:49.502731 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:49.502736 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:49.502761 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:49.502986 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:49.502994 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.504604 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.504611 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:49.506137 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:49.506148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.508237 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.508243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:49.508468 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:49.508544 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 02:52:49.508631 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 02:52:49.508725 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13348,"thread_start_us":13239,"threads_started":1}
I20260620 02:52:49.509189 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:49.488782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59864 (local address 127.0.97.61:41347)
0620 02:52:49.501176 (+ 12394us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:49.501181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:49.502128 (+ 947us) server_negotiation.cc:407] Connection header received
0620 02:52:49.502465 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:49.502473 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:49.502583 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:49.502695 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:49.503151 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.504381 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.506320 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.507717 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.507901 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:49.508147 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:49.508687 (+ 540us) server_negotiation.cc:299] Negotiation successful
0620 02:52:49.508823 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12242,"thread_start_us":12399,"threads_started":1}
I20260620 02:52:49.510221 7166 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260620 02:52:49.510857 7194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1caa9396c123467c8288bc9f054a1c1f. Latest consensus state: current_term: 1 leader_uuid: "1caa9396c123467c8288bc9f054a1c1f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: true } } }
I20260620 02:52:49.511112 7194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [sys.catalog]: This master's current role is: LEARNER
Jun 20 02:52:49 dist-test-slave-8p8c krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923969, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 02:52:49.521265 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:49.507439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41347 (local address 127.0.0.1:59868)
0620 02:52:49.507618 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:49.507639 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:49.507740 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:49.508384 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:49.508387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:49.508786 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:49.509015 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:49.509029 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.510194 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.510200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:49.511358 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:49.511371 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.512053 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.512059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:49.512663 (+ 604us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:49.513270 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:49.513288 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:49.515465 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:49.518845 (+ 3380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:49.518851 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:49.518855 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:49.519233 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:49.519615 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:49.519624 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:49.519626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:49.519749 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:49.520269 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:49.520273 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:49.520421 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 02:52:49.520599 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 02:52:49.520912 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 02:52:49.521302 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:49.507537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59868 (local address 127.0.97.61:41347)
0620 02:52:49.508092 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:49.508097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:49.508118 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:52:49.508165 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:49.508172 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:49.508243 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:49.508347 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:49.509204 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.510052 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.511488 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.511894 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.512033 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:49.515714 (+ 3681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:49.515719 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:49.515722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:49.515753 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:49.518708 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:49.519353 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:49.519360 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:49.519364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:49.519497 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:49.519850 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:49.519857 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:49.519860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:49.520157 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:49.520303 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:49.520690 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 02:52:49.520912 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":292,"threads_started":1}
I20260620 02:52:49.522584 7272 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43075 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:52:49 dist-test-slave-8p8c krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923969, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 02:52:49.535712 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:49.523411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43075 (local address 127.0.0.1:54710)
0620 02:52:49.523635 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:49.523653 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:49.523751 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:49.524030 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:49.524034 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:49.524389 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:49.524548 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:49.524556 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.525497 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.525501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:49.526356 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:49.526437 (+ 81us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.527164 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.527168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:49.527319 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:49.527891 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:49.527908 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:49.529935 (+ 2027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:49.533083 (+ 3148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:49.533088 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:49.533093 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:49.533567 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:49.534095 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:49.534098 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:49.534101 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:49.534267 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:49.534829 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:49.534833 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:49.535094 (+ 261us) client_negotiation.cc:769] Sending connection context
0620 02:52:49.535237 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 02:52:49.535438 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 02:52:49.535924 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:49.523484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54710 (local address 127.0.97.60:43075)
0620 02:52:49.523638 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:49.523642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:49.523765 (+ 123us) server_negotiation.cc:407] Connection header received
0620 02:52:49.523818 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:49.523830 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:49.523884 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:49.524002 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:49.524687 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.525363 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.526570 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.526978 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.527109 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:49.530120 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:49.530124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:49.530127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:49.530156 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:49.532952 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:49.533678 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:49.533683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:49.533685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:49.533836 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:49.534425 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:49.534429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:49.534431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:49.534706 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:49.534815 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:49.535342 (+ 527us) server_negotiation.cc:299] Negotiation successful
0620 02:52:49.535577 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":54}
I20260620 02:52:49.541183 7272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e: Bootstrap starting.
I20260620 02:52:49.544556 7272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:49.545645 7272 log.cc:824] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:49.548808 7272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e: No bootstrap required, opened a new log
I20260620 02:52:49.556792 7272 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43075 } }
I20260620 02:52:49.557175 7272 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:49.557301 7272 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d11130cfec9434094b0340a81f4230e, State: Initialized, Role: FOLLOWER
I20260620 02:52:49.557925 7272 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43075 } }
I20260620 02:52:49.558871 7282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43075 } } }
I20260620 02:52:49.559337 7282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:52:49.559914 7272 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [sys.catalog]: configured and running, proceeding with master startup.
W20260620 02:52:49.569339 7293 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:52:49.569510 7293 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:52:49.573945 7120 master_runner.cc:428] Detected that this master 8d11130cfec9434094b0340a81f4230e is joining an existing cluster
I20260620 02:52:49.574085 7120 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:43075
I20260620 02:52:49.601451 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:49.578006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60494 (local address 127.0.97.62:45057)
0620 02:52:49.578295 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:49.578302 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:49.580888 (+ 2586us) server_negotiation.cc:407] Connection header received
0620 02:52:49.581122 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:49.581129 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:49.581213 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:49.581321 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:49.583387 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.584677 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.589773 (+ 5096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.590703 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.590826 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:49.594090 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:49.594096 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:49.594100 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:49.594137 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:49.598232 (+ 4095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:49.599046 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:49.599051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:49.599055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:49.599197 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:49.599658 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:49.599662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:49.599665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:49.599986 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:49.600111 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:49.600735 (+ 624us) server_negotiation.cc:299] Negotiation successful
0620 02:52:49.601001 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260620 02:52:49.602577 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:49.578068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.0.1:60494)
0620 02:52:49.580749 (+ 2681us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:49.580777 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:49.580963 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:49.581374 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:49.581382 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:49.582302 (+ 920us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:49.583205 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:49.583221 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.585168 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.585176 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:49.589564 (+ 4388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:49.589583 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.591192 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.591202 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:49.591447 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:49.592288 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:49.592327 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:49.593858 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:49.598398 (+ 4540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:49.598421 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:49.598427 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:49.598897 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:49.599333 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:49.599337 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:49.599341 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:49.599516 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:49.600190 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:49.600197 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:49.600397 (+ 200us) client_negotiation.cc:769] Sending connection context
0620 02:52:49.601710 (+ 1313us) client_negotiation.cc:240] Negotiation successful
0620 02:52:49.602058 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2475,"mutex_wait_us":97,"thread_start_us":4503,"threads_started":1}
I20260620 02:52:49.607762 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 6617
I20260620 02:52:49.614675 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:49.578219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54724 (local address 127.0.97.60:43075)
0620 02:52:49.578473 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:49.578482 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:49.583441 (+ 4959us) server_negotiation.cc:407] Connection header received
0620 02:52:49.583637 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:49.583642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:49.583708 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:49.583770 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:49.585172 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.586753 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.589564 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.590265 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.590395 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:49.593210 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:49.593222 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:49.593227 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:49.593260 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:49.600454 (+ 7194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:49.612336 (+ 11882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:49.612342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:49.612345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:49.612518 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:49.612936 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:49.612941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:49.612944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:49.613325 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:49.613481 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:49.614002 (+ 521us) server_negotiation.cc:299] Negotiation successful
0620 02:52:49.614232 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":57}
I20260620 02:52:49.616070 7299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:49.579883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43075 (local address 127.0.0.1:54724)
0620 02:52:49.583356 (+ 3473us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:49.583374 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:49.583521 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:49.583958 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:49.583962 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:49.584632 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:49.584968 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:49.584985 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.586994 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.587000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:49.589396 (+ 2396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:49.589410 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.590700 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.590706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:49.590875 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:49.591642 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:49.591670 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:49.593012 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:49.608013 (+ 15001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:49.608020 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:49.608024 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:49.608504 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:49.612676 (+ 4172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:49.612681 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:49.612685 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:49.612827 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:49.613546 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:49.613550 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:49.613753 (+ 203us) client_negotiation.cc:769] Sending connection context
0620 02:52:49.615295 (+ 1542us) client_negotiation.cc:240] Negotiation successful
0620 02:52:49.615596 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3305,"mutex_wait_us":125,"thread_start_us":472,"threads_started":1}
I20260620 02:52:49.616088 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:49.583661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59878 (local address 127.0.97.61:41347)
0620 02:52:49.583831 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:49.583837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:49.594011 (+ 10174us) server_negotiation.cc:407] Connection header received
0620 02:52:49.594403 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:49.594410 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:49.594488 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:49.594569 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:49.596006 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.597202 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.599664 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.600280 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.600375 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:49.602959 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:49.602964 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:49.602969 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:49.603006 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:49.606415 (+ 3409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:49.607237 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:49.607245 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:49.607248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:49.607408 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:49.607856 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:49.607860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:49.607865 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:49.608176 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:49.608313 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:49.615514 (+ 7201us) server_negotiation.cc:299] Negotiation successful
0620 02:52:49.615755 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 02:52:49.617062 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:49.583949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41347 (local address 127.0.0.1:59878)
0620 02:52:49.593915 (+ 9966us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:49.593935 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:49.594233 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:49.594782 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:49.594792 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:49.595461 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:49.595826 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:49.595840 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.597396 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.597429 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:49.599513 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:49.599523 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.600665 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.600672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:49.600819 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:49.601408 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:49.601435 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:49.602592 (+ 1157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:49.606597 (+ 4005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:49.606622 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:49.606627 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:49.607093 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:49.607551 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:49.607555 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:49.607558 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:49.607732 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:49.614968 (+ 7236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:49.614973 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:49.615185 (+ 212us) client_negotiation.cc:769] Sending connection context
0620 02:52:49.616314 (+ 1129us) client_negotiation.cc:240] Negotiation successful
0620 02:52:49.616635 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9821,"thread_start_us":10082,"threads_started":1}
Jun 20 02:52:49 dist-test-slave-8p8c krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
Jun 20 02:52:49 dist-test-slave-8p8c krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923966, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 02:52:49.632480 7302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:49.608977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49932 (local address 127.0.97.1:43733)
0620 02:52:49.609562 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:49.609570 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:49.609599 (+ 29us) server_negotiation.cc:407] Connection header received
0620 02:52:49.609664 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:49.609677 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:49.610030 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:49.612345 (+ 2315us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:49.612636 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.614673 (+ 2037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.616687 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.617659 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.618659 (+ 1000us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:49.624123 (+ 5464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:49.624138 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:49.624157 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:49.624208 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:49.628486 (+ 4278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:49.629813 (+ 1327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:49.629821 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:49.629824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:49.630025 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:49.630561 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:49.630567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:49.630571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:49.631059 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:49.631386 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:49.631731 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 02:52:49.632019 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":491,"threads_started":1}
I20260620 02:52:49.634039 6740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:49.636936 7303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:49.609250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54732 (local address 127.0.97.60:43075)
0620 02:52:49.615122 (+ 5872us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:49.615129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:49.615156 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:52:49.615241 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:49.615247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:49.615328 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:49.618783 (+ 3455us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:49.619030 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.620638 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.622883 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.623680 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.628141 (+ 4461us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:49.628425 (+ 284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:49.628432 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:49.628437 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:49.628484 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:49.632956 (+ 4472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:49.633846 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:49.633852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:49.633855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:49.634037 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:49.634659 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:49.634665 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:49.634669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:49.635029 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:49.635168 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:49.635897 (+ 729us) server_negotiation.cc:299] Negotiation successful
0620 02:52:49.636258 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5730,"thread_start_us":5550,"threads_started":1}
I20260620 02:52:49.638461 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:49.608582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43075 (local address 127.0.0.1:54732)
0620 02:52:49.608731 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:49.608749 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:49.608848 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:49.617897 (+ 9049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:49.617904 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:49.618454 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:49.618636 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:49.618650 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.620806 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.620815 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:49.622680 (+ 1865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:49.622699 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.623888 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.623898 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:49.624123 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:49.624847 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:49.624874 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:49.627943 (+ 3069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:49.633093 (+ 5150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:49.633099 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:49.633103 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:49.633685 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:49.634285 (+ 600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:49.634290 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:49.634294 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:49.634505 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:49.635239 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:49.635244 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:49.635430 (+ 186us) client_negotiation.cc:769] Sending connection context
0620 02:52:49.637280 (+ 1850us) client_negotiation.cc:240] Negotiation successful
0620 02:52:49.637933 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 02:52:49.640336 6556 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:43075: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8d11130cfec9434094b0340a81f4230e"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 43075
}
attrs {
promote: true
}
}
dest_uuid: "1caa9396c123467c8288bc9f054a1c1f"
cas_config_opid_index: 5
I20260620 02:52:49.641952 6556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: true } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: true } }
I20260620 02:52:49.644546 7166 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [term 1 LEARNER]: Refusing update from remote peer 1caa9396c123467c8288bc9f054a1c1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 02:52:49.645264 7277 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 02:52:49.647009 7247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:49.651685 7247 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [term 1 FOLLOWER]: Refusing update from remote peer 1caa9396c123467c8288bc9f054a1c1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 02:52:49.652812 7277 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8d11130cfec9434094b0340a81f4230e (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: true } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: true } } }
I20260620 02:52:49.658452 7277 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:43075
I20260620 02:52:49.659152 7120 master.cc:537] Master@127.0.97.60:43075 shutting down...
I20260620 02:52:49.660331 7277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1caa9396c123467c8288bc9f054a1c1f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: true } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: true } } }
I20260620 02:52:49.660720 7277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:49.675360 7277 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:52:49.681871 7166 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8d11130cfec9434094b0340a81f4230e (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: true } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: true } } }
I20260620 02:52:49.686866 7194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1caa9396c123467c8288bc9f054a1c1f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: true } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: true } } }
I20260620 02:52:49.687163 7194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [sys.catalog]: This master's current role is: LEARNER
I20260620 02:52:49.687669 7306 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f: attempting to promote NON_VOTER 982c8b2c204d443087f6c9f667437974 to VOTER
I20260620 02:52:49.688295 7306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: false } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: true } }
I20260620 02:52:49.690330 7166 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [term 1 LEARNER]: Refusing update from remote peer 1caa9396c123467c8288bc9f054a1c1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 02:52:49.690973 7277 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 02:52:49.693038 7306 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 982c8b2c204d443087f6c9f667437974 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: false } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: true } } }
I20260620 02:52:49.697196 7120 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:52:49.697729 7120 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:49.697964 7120 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e: stopping tablet replica
I20260620 02:52:49.698700 7166 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 982c8b2c204d443087f6c9f667437974 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: false } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: true } } }
I20260620 02:52:49.704643 7120 master.cc:552] Master@127.0.97.60:43075 shutdown complete.
I20260620 02:52:49.704734 7120 master_runner.cc:305] Clearing existing system tablet
I20260620 02:52:49.704624 7306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1caa9396c123467c8288bc9f054a1c1f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: false } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: true } } }
I20260620 02:52:49.705155 7120 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:49.705243 7120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:49.704986 7305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1caa9396c123467c8288bc9f054a1c1f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: false } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: true } } }
I20260620 02:52:49.705417 7305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:49.708295 7306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:49.710465 7194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1caa9396c123467c8288bc9f054a1c1f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: false } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: true } } }
I20260620 02:52:49.710701 7194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:52:49.713965 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:49.713545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43075 (local address 127.0.0.1:54742)
0620 02:52:49.713685 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:49.713794 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 02:52:49.714246 6542 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f -> Peer 8d11130cfec9434094b0340a81f4230e (127.0.97.60:43075): Couldn't send request to peer 8d11130cfec9434094b0340a81f4230e. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 02:52:49.718621 7120 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.001s sys 0.010s
I20260620 02:52:49.720394 7311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:49.720803 7120 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:49.721271 7120 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260620 02:52:49.721426 7120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-2/wal
uuid: "8d11130cfec9434094b0340a81f4230e"
format_stamp: "Formatted at 2026-06-20 02:52:49 on dist-test-slave-8p8c"
server_key: "27fb06d27785e57cc7edf3094ea49e38"
server_key_iv: "4827dd067c24c2ff6edd27f56adfdb86"
server_key_version: "encryptionkey@0"
I20260620 02:52:49.727070 7120 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:49.734625 7120 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 02:52:49.734894 7120 log.cc:1196] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 02:52:49.735496 7120 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e: Deleting consensus metadata
I20260620 02:52:49.735844 7120 master_runner.cc:315] Copying system tablet from 127.0.97.62:45057
I20260620 02:52:49.741426 7120 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:45057
I20260620 02:52:49.766347 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:49.742495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60500 (local address 127.0.97.62:45057)
0620 02:52:49.742738 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:49.742745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:49.751008 (+ 8263us) server_negotiation.cc:407] Connection header received
0620 02:52:49.751206 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:49.751212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:49.751296 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:49.751411 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:49.752525 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.753637 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.755347 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.756068 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.756207 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:49.758998 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:49.759006 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:49.759012 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:49.759076 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:49.763218 (+ 4142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:49.764204 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:49.764210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:49.764214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:49.764368 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:49.764809 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:49.764814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:49.764817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:49.765164 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:49.765311 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:49.765754 (+ 443us) server_negotiation.cc:299] Negotiation successful
0620 02:52:49.765981 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 02:52:49.766392 7317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:49.742395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.0.1:60500)
0620 02:52:49.750835 (+ 8440us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:49.750872 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:49.751066 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:49.751475 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:49.751481 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:49.752045 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:49.752337 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:49.752355 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.753804 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.753809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:49.755196 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:49.755212 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:49.756250 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:49.756256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:49.756435 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:49.757214 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:49.757236 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:49.758351 (+ 1115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:49.763387 (+ 5036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:49.763393 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:49.763397 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:49.764045 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:49.764498 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:49.764503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:49.764507 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:49.764680 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:49.765299 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:49.765304 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:49.765480 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 02:52:49.765623 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 02:52:49.765928 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8277,"thread_start_us":8148,"threads_started":1}
I20260620 02:52:49.767467 6576 tablet_copy_service.cc:140] P 1caa9396c123467c8288bc9f054a1c1f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8d11130cfec9434094b0340a81f4230e ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:60500)
I20260620 02:52:49.767650 6576 tablet_copy_service.cc:161] P 1caa9396c123467c8288bc9f054a1c1f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8d11130cfec9434094b0340a81f4230e at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:60500: session id = 8d11130cfec9434094b0340a81f4230e-00000000000000000000000000000000
I20260620 02:52:49.771443 6576 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 02:52:49.771927 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 6617
I20260620 02:52:49.773304 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 02:52:49.783840 7120 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e: tablet copy: Starting download of 0 data blocks...
I20260620 02:52:49.784148 7120 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e: tablet copy: Starting download of 1 WAL segments...
I20260620 02:52:49.786520 7120 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 02:52:49.789736 6576 tablet_copy_service.cc:342] P 1caa9396c123467c8288bc9f054a1c1f: Request end of tablet copy session 8d11130cfec9434094b0340a81f4230e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:60500
I20260620 02:52:49.789880 6576 tablet_copy_service.cc:433] P 1caa9396c123467c8288bc9f054a1c1f: ending tablet copy session 8d11130cfec9434094b0340a81f4230e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8d11130cfec9434094b0340a81f4230e
I20260620 02:52:49.797732 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:43733
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45057
--webserver_port=44557
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:45057,127.0.97.61:41347
--tserver_master_addrs=127.0.97.62:45057,127.0.97.61:41347,127.0.97.60:43075 with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:52:49.802265 7120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:49.809388 7318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:49.810688 7320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:49.815757 7120 server_base.cc:1034] running on GCE node
W20260620 02:52:49.822736 7322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:49.823239 7120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:49.823931 7120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:49.825150 7120 hybrid_clock.cc:648] HybridClock initialized: now 1781923969825090 us; error 57 us; skew 500 ppm
I20260620 02:52:49.825398 7120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:49 dist-test-slave-8p8c krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923969, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:49.827629 7120 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260620 02:52:49.833539 7120 webserver.cc:460] Webserver started at http://127.0.97.60:46837/ using document root <none> and password file <none>
I20260620 02:52:49.833879 7120 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:49.833998 7120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:49.837071 7120 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 02:52:49.839077 7329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:49.839522 7120 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 02:52:49.839651 7120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-2/wal
uuid: "8d11130cfec9434094b0340a81f4230e"
format_stamp: "Formatted at 2026-06-20 02:52:49 on dist-test-slave-8p8c"
server_key: "27fb06d27785e57cc7edf3094ea49e38"
server_key_iv: "4827dd067c24c2ff6edd27f56adfdb86"
server_key_version: "encryptionkey@0"
I20260620 02:52:49.839833 7120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:49.868299 7120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:49.872843 7120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:49.907397 7120 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:43075
I20260620 02:52:49.907445 7380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:43075 every 8 connection(s)
I20260620 02:52:49.908492 7120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/master-2/data/info.pb
I20260620 02:52:49.913894 7381 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:52:49.918279 7381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e: Bootstrap starting.
I20260620 02:52:49.936924 7381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:52:49.937469 7381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e: Bootstrap complete.
I20260620 02:52:49.938588 7381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: false } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: true } }
I20260620 02:52:49.938942 7381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d11130cfec9434094b0340a81f4230e, State: Initialized, Role: LEARNER
I20260620 02:52:49.939254 7381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: false } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: true } }
I20260620 02:52:49.940781 7384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: false } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: true } } }
I20260620 02:52:49.941123 7381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:52:49.941344 7384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [sys.catalog]: This master's current role is: LEARNER
I20260620 02:52:49.953408 7395 catalog_manager.cc:1259] Loaded cluster ID: 13341c1efd294e0597c84690d18aca07
I20260620 02:52:49.953540 7395 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e: loading cluster ID for follower catalog manager: success
I20260620 02:52:49.957362 7395 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e: acquiring CA information for follower catalog manager: success
I20260620 02:52:49.960109 7395 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 02:52:50.187906 7319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:50.188221 7319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:50.188371 7319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:50.195729 7319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:50.195870 7319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:50.196035 7319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:52:50.207190 7319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:43733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=44557
--tserver_master_addrs=127.0.97.62:45057,127.0.97.61:41347,127.0.97.60:43075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:50.208531 7319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:50.209949 7319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:50.223935 7400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:50.224017 7401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:50.224279 7403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:50.224901 7319 server_base.cc:1034] running on GCE node
I20260620 02:52:50.225618 7319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:50.226711 7319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:50.228009 7319 hybrid_clock.cc:648] HybridClock initialized: now 1781923970227922 us; error 45 us; skew 500 ppm
I20260620 02:52:50.228355 7319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:50 dist-test-slave-8p8c krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923970, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:50.232221 7319 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 02:52:50.233629 7319 webserver.cc:460] Webserver started at http://127.0.97.1:44557/ using document root <none> and password file <none>
I20260620 02:52:50.234314 7319 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:50.234509 7319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:50.240057 7319 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 02:52:50.243913 7410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:50.245608 7319 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260620 02:52:50.245780 7319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-0/wal
uuid: "33858113a83d4f2da92c8f2364a93c83"
format_stamp: "Formatted at 2026-06-20 02:52:47 on dist-test-slave-8p8c"
server_key: "c5a1e83a08a8dc3a81241570b63fb197"
server_key_iv: "73118d98561e5369c037bd3c56a85948"
server_key_version: "encryptionkey@0"
I20260620 02:52:50.246850 7319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:50.260566 7319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:50.264860 7319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:50.265161 7319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:50.266160 7319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:52:50.267700 7319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:50.267784 7319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:50.267937 7319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:50.268026 7319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:50.278002 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:50.269219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54752 (local address 127.0.97.60:43075)
0620 02:52:50.270530 (+ 1311us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:50.270536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:50.270559 (+ 23us) server_negotiation.cc:407] Connection header received
0620 02:52:50.270631 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:50.270636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:50.271124 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:50.271263 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:50.271652 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:50.273299 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:50.275294 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:50.276926 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:50.277109 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:50.277403 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:50.277547 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 02:52:50.277699 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1123,"thread_start_us":996,"threads_started":1}
I20260620 02:52:50.277999 7418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:50.269066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43075 (local address 127.0.0.1:54752)
0620 02:52:50.269854 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:50.269875 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:50.270016 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:50.271307 (+ 1291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:50.271311 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:50.271335 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:50.271493 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:50.271501 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:50.273449 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:50.273457 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:50.275116 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:50.275130 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:50.277153 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:50.277161 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:50.277389 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:50.277416 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 02:52:50.277527 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 02:52:50.277663 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":575,"threads_started":1}
I20260620 02:52:50.278982 7356 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 02:52:50.279807 7384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1caa9396c123467c8288bc9f054a1c1f. Latest consensus state: current_term: 1 leader_uuid: "1caa9396c123467c8288bc9f054a1c1f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: false } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: true } } }
I20260620 02:52:50.280097 7384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [sys.catalog]: This master's current role is: LEARNER
I20260620 02:52:50.308727 7319 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43733
I20260620 02:52:50.308779 7524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43733 every 8 connection(s)
I20260620 02:52:50.310250 7319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:52:50.316537 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 7319
I20260620 02:52:50.316897 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 6751
Jun 20 02:52:50 dist-test-slave-8p8c krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923970, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 20 02:52:50 dist-test-slave-8p8c krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
Jun 20 02:52:50 dist-test-slave-8p8c krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923970, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 20 02:52:50 dist-test-slave-8p8c krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923970, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:52:50.340149 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:50.313853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51023 (local address 127.0.97.60:43075)
0620 02:52:50.314048 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:50.314055 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:50.315126 (+ 1071us) server_negotiation.cc:407] Connection header received
0620 02:52:50.319058 (+ 3932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:50.319065 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:50.319151 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:50.319221 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:50.322286 (+ 3065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:50.323412 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:50.326663 (+ 3251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:50.327331 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:50.327429 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:50.332291 (+ 4862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:50.332298 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:50.332303 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:50.332351 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:50.336216 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:50.337203 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:50.337213 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:50.337218 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:50.337369 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:50.337933 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:50.337939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:50.337943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:50.338314 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:50.338498 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:50.339555 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0620 02:52:50.339796 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 02:52:50.341549 7529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:50.313184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53901 (local address 127.0.97.61:41347)
0620 02:52:50.313731 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:50.313738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:50.315352 (+ 1614us) server_negotiation.cc:407] Connection header received
0620 02:52:50.319138 (+ 3786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:50.319144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:50.319229 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:50.319329 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:50.322287 (+ 2958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:50.323415 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:50.327587 (+ 4172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:50.328305 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:50.328421 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:50.333529 (+ 5108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:50.333537 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:50.333543 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:50.333590 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:50.337485 (+ 3895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:50.338314 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:50.338320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:50.338325 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:50.338518 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:50.338941 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:50.338946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:50.338948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:50.339309 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:50.339439 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:50.340897 (+ 1458us) server_negotiation.cc:299] Negotiation successful
0620 02:52:50.341109 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":216,"threads_started":1}
I20260620 02:52:50.342509 7535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:50.317977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36064 (local address 127.0.97.2:36091)
0620 02:52:50.318856 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:50.318865 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:50.318894 (+ 29us) server_negotiation.cc:407] Connection header received
0620 02:52:50.318971 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:50.318979 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:50.319219 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:50.319385 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:50.321609 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:50.323228 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:50.324908 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:50.325717 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:50.325845 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:50.332730 (+ 6885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:50.332748 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:50.332768 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:50.332810 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:50.337640 (+ 4830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:50.338617 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:50.338626 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:50.338630 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:50.338787 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:50.339254 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:50.339258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:50.339262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:50.339723 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:50.339988 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:50.340947 (+ 959us) server_negotiation.cc:299] Negotiation successful
0620 02:52:50.341189 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":555,"threads_started":1}
I20260620 02:52:50.342475 7530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:50.313804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43075 (local address 127.0.97.1:51023)
0620 02:52:50.314869 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:50.314920 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:50.318722 (+ 3802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:50.319703 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:50.319714 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:50.321028 (+ 1314us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:50.322077 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:50.322102 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:50.323606 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:50.323611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:50.326466 (+ 2855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:50.326491 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:50.327746 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:50.327753 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:50.328038 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:50.328813 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:50.328850 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:50.331989 (+ 3139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:50.336388 (+ 4399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:50.336397 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:50.336418 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:50.337058 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:50.337533 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:50.337541 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:50.337547 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:50.337793 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:50.338600 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:50.338610 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:50.339099 (+ 489us) client_negotiation.cc:769] Sending connection context
0620 02:52:50.340379 (+ 1280us) client_negotiation.cc:240] Negotiation successful
0620 02:52:50.340828 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"mutex_wait_us":260,"spinlock_wait_cycles":65792,"thread_start_us":395,"threads_started":1}
I20260620 02:52:50.342447 7531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:50.313804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41347 (local address 127.0.97.1:53901)
0620 02:52:50.314869 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:50.314921 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:50.318731 (+ 3810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:50.319701 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:50.319715 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:50.321027 (+ 1312us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:50.322101 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:50.322116 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:50.323583 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:50.323588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:50.327407 (+ 3819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:50.327424 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:50.328732 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:50.328741 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:50.329011 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:50.329655 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:50.329688 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:50.333289 (+ 3601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:50.337644 (+ 4355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:50.337649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:50.337652 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:50.338168 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:50.338645 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:50.338650 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:50.338653 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:50.338799 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:50.339510 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:50.339514 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:50.339642 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 02:52:50.339807 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 02:52:50.340369 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"mutex_wait_us":1476,"spinlock_wait_cycles":11008,"thread_start_us":488,"threads_started":1}
I20260620 02:52:50.344302 7525 heartbeater.cc:347] Connected to a master server at 127.0.97.60:43075
I20260620 02:52:50.344333 7527 heartbeater.cc:347] Connected to a master server at 127.0.97.61:41347
I20260620 02:52:50.344424 6874 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:50.344666 7527 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:50.344661 7525 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:50.345583 7527 heartbeater.cc:511] Master 127.0.97.61:41347 requested a full tablet report, sending...
I20260620 02:52:50.345628 7525 heartbeater.cc:511] Master 127.0.97.60:43075 requested a full tablet report, sending...
I20260620 02:52:50.346573 7533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:50.315679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41569 (local address 127.0.97.62:45057)
0620 02:52:50.317623 (+ 1944us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:50.317629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:50.317649 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:52:50.319138 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:50.319143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:50.319221 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:50.319314 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:50.322298 (+ 2984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:50.323476 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:50.327471 (+ 3995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:50.328090 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:50.328211 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:50.334957 (+ 6746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:50.334966 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:50.334986 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:50.335033 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:50.338792 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:50.339587 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:50.339593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:50.339595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:50.339801 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:50.340374 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:50.340378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:50.340381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:50.340897 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:50.341096 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:50.341876 (+ 780us) server_negotiation.cc:299] Negotiation successful
0620 02:52:50.342193 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1827,"thread_start_us":1706,"threads_started":1}
I20260620 02:52:50.348209 7156 ts_manager.cc:194] Registered new tserver with Master: 33858113a83d4f2da92c8f2364a93c83 (127.0.97.1:43733)
I20260620 02:52:50.348742 7346 ts_manager.cc:194] Registered new tserver with Master: 33858113a83d4f2da92c8f2364a93c83 (127.0.97.1:43733)
I20260620 02:52:50.349359 7532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:50.315554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.97.1:41569)
0620 02:52:50.316540 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:50.316563 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:50.318754 (+ 2191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:50.319703 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:50.319714 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:50.321028 (+ 1314us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:50.322077 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:50.322102 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:50.323610 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:50.323614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:50.327282 (+ 3668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:50.327297 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:50.328468 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:50.328473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:50.328626 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:50.329207 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:50.329229 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:50.334726 (+ 5497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:50.338934 (+ 4208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:50.338938 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:50.338941 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:50.339459 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:50.340009 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:50.340014 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:50.340018 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:50.340180 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:50.341173 (+ 993us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:50.341178 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:50.341325 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 02:52:50.342404 (+ 1079us) client_negotiation.cc:240] Negotiation successful
0620 02:52:50.342693 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":1479,"spinlock_wait_cycles":10752,"thread_start_us":293,"threads_started":1}
I20260620 02:52:50.350193 7526 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45057
I20260620 02:52:50.350351 7526 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:50.350971 7526 heartbeater.cc:511] Master 127.0.97.62:45057 requested a full tablet report, sending...
I20260620 02:52:50.352458 6556 ts_manager.cc:194] Re-registered known tserver with Master: 33858113a83d4f2da92c8f2364a93c83 (127.0.97.1:43733)
I20260620 02:52:50.353708 6556 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41569
I20260620 02:52:50.440205 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 6751
I20260620 02:52:50.457322 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:36091
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45057
--webserver_port=32999
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:45057,127.0.97.61:41347
--tserver_master_addrs=127.0.97.62:45057,127.0.97.61:41347,127.0.97.60:43075 with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:50.792508 7537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:50.792805 7537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:50.792958 7537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:50.800436 7537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:50.800554 7537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:50.800734 7537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:52:50.811439 7537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:36091
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=32999
--tserver_master_addrs=127.0.97.62:45057,127.0.97.61:41347,127.0.97.60:43075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:50.813366 7537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:50.814824 7537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:50.828230 7542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:50.828399 7543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:50.828713 7545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:50.829346 7537 server_base.cc:1034] running on GCE node
I20260620 02:52:50.829901 7537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:50.830897 7537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:50.832104 7537 hybrid_clock.cc:648] HybridClock initialized: now 1781923970832040 us; error 58 us; skew 500 ppm
I20260620 02:52:50.832386 7537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:50 dist-test-slave-8p8c krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923970, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:50.835732 7537 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 02:52:50.837289 7537 webserver.cc:460] Webserver started at http://127.0.97.2:32999/ using document root <none> and password file <none>
I20260620 02:52:50.837945 7537 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:50.838057 7537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:50.845005 7537 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 02:52:50.847025 7306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1caa9396c123467c8288bc9f054a1c1f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: false } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: true } } }
I20260620 02:52:50.847421 7306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:50.848573 7306 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f: attempting to promote NON_VOTER 8d11130cfec9434094b0340a81f4230e to VOTER
I20260620 02:52:50.849146 7555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:50.849242 7306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: false } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: false } }
I20260620 02:52:50.851127 7537 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260620 02:52:50.851543 7356 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [term 1 LEARNER]: Refusing update from remote peer 1caa9396c123467c8288bc9f054a1c1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 02:52:50.851586 7166 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [term 1 FOLLOWER]: Refusing update from remote peer 1caa9396c123467c8288bc9f054a1c1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 02:52:50.852134 7553 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 02:52:50.851303 7537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-1/wal
uuid: "66f25b3d96fa40f294be341e1266df2d"
format_stamp: "Formatted at 2026-06-20 02:52:47 on dist-test-slave-8p8c"
server_key: "6de2c406956d0933721b3fb94536d981"
server_key_iv: "bf1500b6d9704b6f897c39f33fb9ca0c"
server_key_version: "encryptionkey@0"
I20260620 02:52:50.852558 7552 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 02:52:50.852663 7537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:50.855365 7553 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8d11130cfec9434094b0340a81f4230e (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: false } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: false } } }
I20260620 02:52:50.856022 7166 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8d11130cfec9434094b0340a81f4230e (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: false } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: false } } }
I20260620 02:52:50.858531 7553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1caa9396c123467c8288bc9f054a1c1f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: false } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: false } } }
I20260620 02:52:50.858779 7553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:50.858999 7553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1caa9396c123467c8288bc9f054a1c1f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: false } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: false } } }
I20260620 02:52:50.859172 7553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:50.860580 7560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1caa9396c123467c8288bc9f054a1c1f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: false } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: false } } }
I20260620 02:52:50.860862 7560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:52:50.860965 7553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1caa9396c123467c8288bc9f054a1c1f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: false } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: false } } }
I20260620 02:52:50.861220 7553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:50.861852 7356 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8d11130cfec9434094b0340a81f4230e (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: false } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: false } } }
I20260620 02:52:50.865393 7564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1caa9396c123467c8288bc9f054a1c1f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: false } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: false } } }
I20260620 02:52:50.865726 7564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:52:50.884078 7537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:50.888710 7537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:50.889021 7537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:50.890061 7537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:52:50.891523 7537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:50.891645 7537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:50.891783 7537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:50.891872 7537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:50.929560 7537 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36091
I20260620 02:52:50.929649 7672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36091 every 8 connection(s)
I20260620 02:52:50.931097 7537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:52:50.939528 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 7537
I20260620 02:52:50.939857 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 6885
Jun 20 02:52:50 dist-test-slave-8p8c krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923970, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 20 02:52:50 dist-test-slave-8p8c krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923970, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 02:52:50.978451 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:50.934767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59779 (local address 127.0.97.60:43075)
0620 02:52:50.954800 (+ 20033us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:50.954810 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:50.954837 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:52:50.954911 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:50.954917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:50.955041 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:50.955177 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:50.958306 (+ 3129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:50.959694 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:50.963880 (+ 4186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:50.964561 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:50.964659 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:50.970050 (+ 5391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:50.970060 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:50.970069 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:50.970119 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:50.974104 (+ 3985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:50.975236 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:50.975245 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:50.975251 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:50.975407 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:50.976069 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:50.976075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:50.976078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:50.976434 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:50.976574 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:50.977614 (+ 1040us) server_negotiation.cc:299] Negotiation successful
0620 02:52:50.977848 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19847,"thread_start_us":20572,"threads_started":1}
Jun 20 02:52:50 dist-test-slave-8p8c krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260620 02:52:50.980010 7680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:50.935067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43075 (local address 127.0.97.2:59779)
0620 02:52:50.945833 (+ 10766us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:50.945939 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:50.950961 (+ 5022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:50.955887 (+ 4926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:50.955906 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:50.957002 (+ 1096us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:50.958047 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:50.958079 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:50.959951 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:50.959960 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:50.963688 (+ 3728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:50.963704 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:50.965095 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:50.965102 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:50.965414 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:50.966402 (+ 988us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:50.966454 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:50.969747 (+ 3293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:50.974276 (+ 4529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:50.974286 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:50.974309 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:50.975088 (+ 779us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:50.975602 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:50.975608 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:50.975619 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:50.975911 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:50.976684 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:50.976692 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:50.977226 (+ 534us) client_negotiation.cc:769] Sending connection context
0620 02:52:50.977529 (+ 303us) client_negotiation.cc:240] Negotiation successful
0620 02:52:50.978082 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10364,"thread_start_us":38839,"threads_started":1}
I20260620 02:52:50.982093 7673 heartbeater.cc:347] Connected to a master server at 127.0.97.60:43075
I20260620 02:52:50.982504 7673 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:50.983776 7673 heartbeater.cc:511] Master 127.0.97.60:43075 requested a full tablet report, sending...
I20260620 02:52:50.985781 7346 ts_manager.cc:194] Registered new tserver with Master: 66f25b3d96fa40f294be341e1266df2d (127.0.97.2:36091)
I20260620 02:52:50.986860 7677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:50.934298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45893 (local address 127.0.97.61:41347)
0620 02:52:50.943625 (+ 9327us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:50.943635 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:50.961271 (+ 17636us) server_negotiation.cc:407] Connection header received
0620 02:52:50.962559 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:50.962569 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:50.962700 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:50.962799 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:50.966298 (+ 3499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:50.967870 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:50.970602 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:50.971979 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:50.972424 (+ 445us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:50.979209 (+ 6785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:50.979218 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:50.979225 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:50.979290 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:50.983396 (+ 4106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:50.984352 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:50.984357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:50.984361 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:50.984522 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:50.985031 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:50.985038 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:50.985044 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:50.985433 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:50.985573 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:50.986224 (+ 651us) server_negotiation.cc:299] Negotiation successful
0620 02:52:50.986497 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8666,"thread_start_us":18239,"threads_started":1}
I20260620 02:52:50.987630 7684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:50.935067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41347 (local address 127.0.97.2:45893)
0620 02:52:50.961127 (+ 26060us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:50.961154 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:50.962178 (+ 1024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:50.963691 (+ 1513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:50.963700 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:50.964552 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:50.965018 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:50.965037 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:50.968168 (+ 3131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:50.968176 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:50.970398 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:50.970416 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:50.972823 (+ 2407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:50.972832 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:50.973092 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:50.974647 (+ 1555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:50.974672 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:50.978986 (+ 4314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:50.983614 (+ 4628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:50.983619 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:50.983627 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:50.984200 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:50.984663 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:50.984667 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:50.984672 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:50.984891 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:50.985649 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:50.985654 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:50.985931 (+ 277us) client_negotiation.cc:769] Sending connection context
0620 02:52:50.986860 (+ 929us) client_negotiation.cc:240] Negotiation successful
0620 02:52:50.987221 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25825,"mutex_wait_us":43,"spinlock_wait_cycles":42752,"thread_start_us":10064,"threads_started":1}
Jun 20 02:52:50 dist-test-slave-8p8c krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923970, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:52:50.988448 7675 heartbeater.cc:347] Connected to a master server at 127.0.97.61:41347
I20260620 02:52:50.988596 7675 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:50.989070 7675 heartbeater.cc:511] Master 127.0.97.61:41347 requested a full tablet report, sending...
I20260620 02:52:50.989869 7682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:50.941793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54904 (local address 127.0.97.3:43517)
0620 02:52:50.964743 (+ 22950us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:50.964754 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:50.964787 (+ 33us) server_negotiation.cc:407] Connection header received
0620 02:52:50.964855 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:50.964866 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:50.965170 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:50.965337 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:50.967606 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:50.969994 (+ 2388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:50.972125 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:50.973063 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:50.973199 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:50.980999 (+ 7800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:50.981026 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:50.981043 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:50.981084 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:50.986681 (+ 5597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:50.987554 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:50.987563 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:50.987567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:50.987723 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:50.988183 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:50.988188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:50.988192 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:50.988632 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:50.988910 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:50.989217 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 02:52:50.989467 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22699,"thread_start_us":28297,"threads_started":1}
I20260620 02:52:50.990217 7156 ts_manager.cc:194] Registered new tserver with Master: 66f25b3d96fa40f294be341e1266df2d (127.0.97.2:36091)
I20260620 02:52:50.991194 7008 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:50.995648 7679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:50.953020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.97.2:40969)
0620 02:52:50.974788 (+ 21768us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:50.974815 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:50.975033 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:50.975648 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:50.975653 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:50.976330 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:50.976559 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:50.976567 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:50.979073 (+ 2506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:50.979079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:50.981347 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:50.981366 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:50.983137 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:50.983144 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:50.983349 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:50.984930 (+ 1581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:50.984955 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:50.988489 (+ 3534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:50.992338 (+ 3849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:50.992343 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:50.992348 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:50.992831 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:50.993275 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:50.993282 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:50.993288 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:50.993439 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:50.994134 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:50.994140 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:50.994296 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 02:52:50.994450 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 02:52:50.994753 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21219,"mutex_wait_us":1026,"thread_start_us":8261,"threads_started":1}
I20260620 02:52:50.996541 7683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:50.947033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40969 (local address 127.0.97.62:45057)
0620 02:52:50.965666 (+ 18633us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:50.965675 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:50.974898 (+ 9223us) server_negotiation.cc:407] Connection header received
0620 02:52:50.975244 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:50.975250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:50.975332 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:50.975415 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:50.976723 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:50.978135 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:50.981519 (+ 3384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:50.982253 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:50.982381 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:50.988778 (+ 6397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:50.988785 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:50.988789 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:50.988837 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:50.992205 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:50.992986 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:50.992992 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:50.992997 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:50.993145 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:50.993560 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:50.993565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:50.993568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:50.993927 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:50.994075 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:50.995969 (+ 1894us) server_negotiation.cc:299] Negotiation successful
0620 02:52:50.996216 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18457,"thread_start_us":18637,"threads_started":1}
I20260620 02:52:50.997241 7674 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45057
I20260620 02:52:50.997381 7674 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:50.997824 7674 heartbeater.cc:511] Master 127.0.97.62:45057 requested a full tablet report, sending...
I20260620 02:52:50.998848 6556 ts_manager.cc:194] Re-registered known tserver with Master: 66f25b3d96fa40f294be341e1266df2d (127.0.97.2:36091)
I20260620 02:52:50.999728 6556 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40969
I20260620 02:52:51.077379 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 6885
I20260620 02:52:51.093521 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:43517
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45057
--webserver_port=35109
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:45057,127.0.97.61:41347
--tserver_master_addrs=127.0.97.62:45057,127.0.97.61:41347,127.0.97.60:43075 with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:52:51.357791 7526 heartbeater.cc:503] Master 127.0.97.62:45057 was elected leader, sending a full tablet report...
W20260620 02:52:51.447038 7686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:51.447362 7686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:51.447523 7686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:51.454955 7686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:51.455109 7686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:51.455281 7686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:52:51.466018 7686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:43517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=35109
--tserver_master_addrs=127.0.97.62:45057,127.0.97.61:41347,127.0.97.60:43075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:51.467600 7686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:51.469161 7686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:51.483938 7691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:51.484004 7692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:51.484040 7694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:51.484644 7686 server_base.cc:1034] running on GCE node
I20260620 02:52:51.485212 7686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:51.486231 7686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:51.487465 7686 hybrid_clock.cc:648] HybridClock initialized: now 1781923971487422 us; error 65 us; skew 500 ppm
I20260620 02:52:51.487845 7686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:51 dist-test-slave-8p8c krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923971, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:51.491349 7686 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 02:52:51.493005 7686 webserver.cc:460] Webserver started at http://127.0.97.3:35109/ using document root <none> and password file <none>
I20260620 02:52:51.493767 7686 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:51.493942 7686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:51.501973 7686 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 02:52:51.506138 7701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:51.508327 7686 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260620 02:52:51.508502 7686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-2/wal
uuid: "56dec7a4e0b8482aa22edc244ee37c51"
format_stamp: "Formatted at 2026-06-20 02:52:48 on dist-test-slave-8p8c"
server_key: "49f44d5943586e36e36e5c05610a9a74"
server_key_iv: "b9d350a1cc57643abb4e1d7548e0c355"
server_key_version: "encryptionkey@0"
I20260620 02:52:51.509500 7686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:51.532763 7686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:51.539876 7686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:51.540256 7686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:51.541898 7686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:52:51.544265 7686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:51.544415 7686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:51.544592 7686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:51.544739 7686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:51.580091 7686 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43517
I20260620 02:52:51.580185 7813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43517 every 8 connection(s)
I20260620 02:52:51.581535 7686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:52:51.585829 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 7686
Jun 20 02:52:51 dist-test-slave-8p8c krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923971, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 02:52:51.623939 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:51.585920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58237 (local address 127.0.97.60:43075)
0620 02:52:51.594132 (+ 8212us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:51.594142 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:51.600769 (+ 6627us) server_negotiation.cc:407] Connection header received
0620 02:52:51.601598 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:51.601603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:51.601695 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:51.601782 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:51.604898 (+ 3116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.606194 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.608977 (+ 2783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.609616 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.609706 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:51.615468 (+ 5762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:51.615476 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:51.615481 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:51.615533 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:51.619771 (+ 4238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:51.620887 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:51.620892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:51.620895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:51.621063 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:51.621671 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:51.621678 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:51.621684 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:51.622031 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:51.622153 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:51.623287 (+ 1134us) server_negotiation.cc:299] Negotiation successful
0620 02:52:51.623551 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7946,"thread_start_us":7804,"threads_started":1}
I20260620 02:52:51.626451 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:51.585273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43075 (local address 127.0.97.3:58237)
0620 02:52:51.600592 (+ 15319us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:51.600611 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:51.601461 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:51.602192 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:51.602205 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:51.603556 (+ 1351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:51.604649 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:51.604680 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.606413 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.606422 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:51.608821 (+ 2399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:51.608835 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.610040 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.610046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:51.610334 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:51.611393 (+ 1059us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:51.611435 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:51.615178 (+ 3743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:51.620010 (+ 4832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:51.620022 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:51.620047 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:51.620747 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:51.621236 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:51.621243 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:51.621250 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:51.621512 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:51.622237 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:51.622245 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:51.622796 (+ 551us) client_negotiation.cc:769] Sending connection context
0620 02:52:51.624212 (+ 1416us) client_negotiation.cc:240] Negotiation successful
0620 02:52:51.624789 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15165,"mutex_wait_us":72,"thread_start_us":15293,"threads_started":1}
I20260620 02:52:51.628191 7814 heartbeater.cc:347] Connected to a master server at 127.0.97.60:43075
I20260620 02:52:51.628958 7814 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:51.630023 7814 heartbeater.cc:511] Master 127.0.97.60:43075 requested a full tablet report, sending...
I20260620 02:52:51.632006 7346 ts_manager.cc:194] Registered new tserver with Master: 56dec7a4e0b8482aa22edc244ee37c51 (127.0.97.3:43517)
Jun 20 02:52:51 dist-test-slave-8p8c krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923971, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 20 02:52:51 dist-test-slave-8p8c krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923971, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:52:51.648036 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:51.585272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41347 (local address 127.0.97.3:51865)
0620 02:52:51.595890 (+ 10618us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:51.595960 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:51.601118 (+ 5158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:51.625933 (+ 24815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:51.625938 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:51.626442 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:51.626707 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:51.626724 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.629448 (+ 2724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.629458 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:51.631630 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:51.631662 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.632754 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.632764 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:51.633086 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:51.634609 (+ 1523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:51.634633 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:51.639867 (+ 5234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:51.644149 (+ 4282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:51.644156 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:51.644161 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:51.644781 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:51.645770 (+ 989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:51.645780 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:51.645786 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:51.646005 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:51.646785 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:51.646791 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:51.646978 (+ 187us) client_negotiation.cc:769] Sending connection context
0620 02:52:51.647152 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 02:52:51.647508 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10274,"mutex_wait_us":4,"thread_start_us":10004,"threads_started":1}
I20260620 02:52:51.649163 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:51.584617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51865 (local address 127.0.97.61:41347)
0620 02:52:51.625512 (+ 40895us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:51.625517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:51.625537 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:52:51.625597 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:51.625603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:51.625675 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:51.626866 (+ 1191us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:51.627095 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.628351 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.631791 (+ 3440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.632476 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.636154 (+ 3678us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:51.640122 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:51.640129 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:51.640133 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:51.640176 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:51.644003 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:51.645162 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:51.645169 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:51.645176 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:51.645357 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:51.646153 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:51.646161 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:51.646166 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:51.646544 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:51.646718 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:51.648541 (+ 1823us) server_negotiation.cc:299] Negotiation successful
0620 02:52:51.648764 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40764,"thread_start_us":41143,"threads_started":1}
I20260620 02:52:51.650525 7816 heartbeater.cc:347] Connected to a master server at 127.0.97.61:41347
I20260620 02:52:51.650949 7816 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:51.651477 7816 heartbeater.cc:511] Master 127.0.97.61:41347 requested a full tablet report, sending...
I20260620 02:52:51.652894 7156 ts_manager.cc:194] Registered new tserver with Master: 56dec7a4e0b8482aa22edc244ee37c51 (127.0.97.3:43517)
I20260620 02:52:51.656916 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:51.634764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54754 (local address 127.0.97.60:43075)
0620 02:52:51.634948 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:51.634955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:51.637540 (+ 2585us) server_negotiation.cc:407] Connection header received
0620 02:52:51.637784 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:51.637792 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:51.637884 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:51.637978 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:51.640474 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.641603 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.643848 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.644649 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.644753 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:51.649909 (+ 5156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:51.649916 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:51.649922 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:51.649966 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:51.653456 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:51.654525 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:51.654530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:51.654535 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:51.654687 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:51.655142 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:51.655147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:51.655150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:51.655486 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:51.655719 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:51.656315 (+ 596us) server_negotiation.cc:299] Negotiation successful
0620 02:52:51.656553 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 02:52:51.658344 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:51.597157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.97.3:40855)
0620 02:52:51.629359 (+ 32202us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:51.629384 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:51.629564 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:51.630111 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:51.630115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:51.630800 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:51.631183 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:51.631199 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.632505 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.632510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:51.634456 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:51.634475 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.635712 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.635720 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:51.635954 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:51.636654 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:51.636675 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:51.648521 (+ 11846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:51.654862 (+ 6341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:51.654868 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:51.654873 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:51.655399 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:51.655830 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:51.655836 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:51.655840 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:51.656012 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:51.656894 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:51.656899 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:51.657086 (+ 187us) client_negotiation.cc:769] Sending connection context
0620 02:52:51.657197 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 02:52:51.657502 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32036,"mutex_wait_us":367,"thread_start_us":31896,"threads_started":1}
I20260620 02:52:51.659343 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:51.597123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40855 (local address 127.0.97.62:45057)
0620 02:52:51.623460 (+ 26337us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:51.623468 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:51.629464 (+ 5996us) server_negotiation.cc:407] Connection header received
0620 02:52:51.629721 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:51.629727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:51.629806 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:51.629890 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:51.631355 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.632355 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.634644 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.635324 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.635462 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:51.648744 (+ 13282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:51.648751 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:51.648755 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:51.648791 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:51.654707 (+ 5916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:51.655554 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:51.655559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:51.655563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:51.655701 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:51.656145 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:51.656150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:51.656153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:51.656474 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:51.658726 (+ 2252us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:51.658839 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 02:52:51.658996 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26193,"thread_start_us":26353,"threads_started":1}
I20260620 02:52:51.660295 7815 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45057
I20260620 02:52:51.660454 7815 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:51.660882 7815 heartbeater.cc:511] Master 127.0.97.62:45057 requested a full tablet report, sending...
I20260620 02:52:51.661791 6556 ts_manager.cc:194] Re-registered known tserver with Master: 56dec7a4e0b8482aa22edc244ee37c51 (127.0.97.3:43517)
I20260620 02:52:51.662935 6556 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40855
I20260620 02:52:51.667042 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:51.636170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60510 (local address 127.0.97.62:45057)
0620 02:52:51.648042 (+ 11872us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:51.648050 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:51.648070 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:52:51.648132 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:51.648137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:51.648225 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:51.650610 (+ 2385us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:51.650803 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.652149 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.655194 (+ 3045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.655850 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.656563 (+ 713us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:51.660549 (+ 3986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:51.660554 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:51.660559 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:51.660593 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:51.664171 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:51.664914 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:51.664920 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:51.664925 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:51.665059 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:51.665446 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:51.665450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:51.665453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:51.665776 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:51.666025 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:51.666413 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 02:52:51.666762 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11717,"spinlock_wait_cycles":5376,"thread_start_us":11693,"threads_started":1}
I20260620 02:52:51.668169 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:51.638494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59894 (local address 127.0.97.61:41347)
0620 02:52:51.647858 (+ 9364us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:51.647864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:51.649091 (+ 1227us) server_negotiation.cc:407] Connection header received
0620 02:52:51.649148 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:51.649158 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:51.649235 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:51.649338 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:51.652696 (+ 3358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.654013 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.656006 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.656636 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.656762 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:51.662038 (+ 5276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:51.662045 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:51.662051 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:51.662091 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:51.665253 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:51.666027 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:51.666034 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:51.666037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:51.666205 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:51.666654 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:51.666659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:51.666662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:51.666949 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:51.667168 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:51.667547 (+ 379us) server_negotiation.cc:299] Negotiation successful
0620 02:52:51.667766 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9229,"thread_start_us":9098,"threads_started":1}
I20260620 02:52:51.671116 6556 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:52:51.678670 6556 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:52:51.718070 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:51.695986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49936 (local address 127.0.97.1:43733)
0620 02:52:51.707897 (+ 11911us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:51.707910 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:51.708538 (+ 628us) server_negotiation.cc:407] Connection header received
0620 02:52:51.708833 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:51.708843 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:51.709086 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:51.709248 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:51.709979 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.711289 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.714460 (+ 3171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.717011 (+ 2551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.717136 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:51.717345 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:51.717716 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 02:52:51.717859 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11698,"thread_start_us":11572,"threads_started":1}
I20260620 02:52:51.718106 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:51.695740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43733 (local address 127.0.0.1:49936)
0620 02:52:51.708392 (+ 12652us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:51.708422 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:51.708594 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:51.709271 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:51.709278 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:51.709311 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:51.709749 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:51.709763 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.711430 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.711439 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:51.714123 (+ 2684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:51.714132 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.717354 (+ 3222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.717362 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:51.717526 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:51.717558 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 02:52:51.717655 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 02:52:51.717784 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12497,"thread_start_us":18227,"threads_started":1}
I20260620 02:52:51.721916 7460 tablet_service.cc:1467] Processing CreateTablet for tablet 34fd838245f34dd586f2e5772fa3c840 (DEFAULT_TABLE table=test-table [id=f0a9678eaeaf42a4a3ec219b873ccd79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:51.722594 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:51.714561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36066 (local address 127.0.97.2:36091)
0620 02:52:51.715730 (+ 1169us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:51.715743 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:51.715764 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:52:51.715836 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:51.715841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:51.716067 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:51.716251 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:51.717258 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.718643 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.720443 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.721278 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.721423 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:51.721645 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:51.722131 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 02:52:51.722290 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":979,"thread_start_us":875,"threads_started":1}
I20260620 02:52:51.722970 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:51.714292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36091 (local address 127.0.0.1:36066)
0620 02:52:51.715394 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:51.715411 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:51.715521 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:51.716428 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:51.716436 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:51.716461 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:51.717079 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:51.717091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.718762 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.718770 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:51.720289 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:51.720303 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.721768 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.721775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:51.721936 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:51.721968 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 02:52:51.722061 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 02:52:51.722186 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"thread_start_us":849,"threads_started":1}
I20260620 02:52:51.723752 7460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34fd838245f34dd586f2e5772fa3c840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:51.724541 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:51.696753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43517 (local address 127.0.0.1:54914)
0620 02:52:51.713758 (+ 17005us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:51.713776 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:51.713894 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:51.715655 (+ 1761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:51.715659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:51.715684 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:51.715923 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:51.715929 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.719203 (+ 3274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.719207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:51.721240 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:51.721253 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.723822 (+ 2569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.723828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:51.724002 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:51.724031 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 02:52:51.724121 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 02:52:51.724238 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16869,"thread_start_us":11324,"threads_started":1}
I20260620 02:52:51.724562 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:51.697014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54914 (local address 127.0.97.3:43517)
0620 02:52:51.707153 (+ 10139us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:51.707166 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:51.713893 (+ 6727us) server_negotiation.cc:407] Connection header received
0620 02:52:51.714096 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:51.714107 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:51.714405 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:51.714650 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:51.716107 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.717590 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.721393 (+ 3803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.723373 (+ 1980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.723512 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:51.723647 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:51.724198 (+ 551us) server_negotiation.cc:299] Negotiation successful
0620 02:52:51.724344 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9883,"thread_start_us":10168,"threads_started":1}
I20260620 02:52:51.727711 7749 tablet_service.cc:1467] Processing CreateTablet for tablet 34fd838245f34dd586f2e5772fa3c840 (DEFAULT_TABLE table=test-table [id=f0a9678eaeaf42a4a3ec219b873ccd79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:51.728804 7749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34fd838245f34dd586f2e5772fa3c840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:51.731453 7608 tablet_service.cc:1467] Processing CreateTablet for tablet 34fd838245f34dd586f2e5772fa3c840 (DEFAULT_TABLE table=test-table [id=f0a9678eaeaf42a4a3ec219b873ccd79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:51.732666 7608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34fd838245f34dd586f2e5772fa3c840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:51.736192 7847 tablet_bootstrap.cc:492] T 34fd838245f34dd586f2e5772fa3c840 P 33858113a83d4f2da92c8f2364a93c83: Bootstrap starting.
I20260620 02:52:51.739055 7847 tablet_bootstrap.cc:654] T 34fd838245f34dd586f2e5772fa3c840 P 33858113a83d4f2da92c8f2364a93c83: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:51.739992 7847 log.cc:824] T 34fd838245f34dd586f2e5772fa3c840 P 33858113a83d4f2da92c8f2364a93c83: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:51.744463 7847 tablet_bootstrap.cc:492] T 34fd838245f34dd586f2e5772fa3c840 P 33858113a83d4f2da92c8f2364a93c83: No bootstrap required, opened a new log
I20260620 02:52:51.744696 7849 tablet_bootstrap.cc:492] T 34fd838245f34dd586f2e5772fa3c840 P 56dec7a4e0b8482aa22edc244ee37c51: Bootstrap starting.
I20260620 02:52:51.744764 7847 ts_tablet_manager.cc:1397] T 34fd838245f34dd586f2e5772fa3c840 P 33858113a83d4f2da92c8f2364a93c83: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 02:52:51.747057 7850 tablet_bootstrap.cc:492] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d: Bootstrap starting.
I20260620 02:52:51.748106 7849 tablet_bootstrap.cc:654] T 34fd838245f34dd586f2e5772fa3c840 P 56dec7a4e0b8482aa22edc244ee37c51: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:51.749272 7849 log.cc:824] T 34fd838245f34dd586f2e5772fa3c840 P 56dec7a4e0b8482aa22edc244ee37c51: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:51.749723 7850 tablet_bootstrap.cc:654] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:51.751078 7850 log.cc:824] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:51.753687 7850 tablet_bootstrap.cc:492] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d: No bootstrap required, opened a new log
I20260620 02:52:51.753964 7850 ts_tablet_manager.cc:1397] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260620 02:52:51.757169 7849 tablet_bootstrap.cc:492] T 34fd838245f34dd586f2e5772fa3c840 P 56dec7a4e0b8482aa22edc244ee37c51: No bootstrap required, opened a new log
I20260620 02:52:51.757455 7849 ts_tablet_manager.cc:1397] T 34fd838245f34dd586f2e5772fa3c840 P 56dec7a4e0b8482aa22edc244ee37c51: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.010s
I20260620 02:52:51.757575 7847 raft_consensus.cc:348] T 34fd838245f34dd586f2e5772fa3c840 P 33858113a83d4f2da92c8f2364a93c83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33858113a83d4f2da92c8f2364a93c83" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43733 } } peers { permanent_uuid: "66f25b3d96fa40f294be341e1266df2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36091 } } peers { permanent_uuid: "56dec7a4e0b8482aa22edc244ee37c51" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43517 } }
I20260620 02:52:51.758107 7847 raft_consensus.cc:374] T 34fd838245f34dd586f2e5772fa3c840 P 33858113a83d4f2da92c8f2364a93c83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:51.758209 7847 raft_consensus.cc:729] T 34fd838245f34dd586f2e5772fa3c840 P 33858113a83d4f2da92c8f2364a93c83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33858113a83d4f2da92c8f2364a93c83, State: Initialized, Role: FOLLOWER
I20260620 02:52:51.758858 7847 consensus_queue.cc:260] T 34fd838245f34dd586f2e5772fa3c840 P 33858113a83d4f2da92c8f2364a93c83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33858113a83d4f2da92c8f2364a93c83" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43733 } } peers { permanent_uuid: "66f25b3d96fa40f294be341e1266df2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36091 } } peers { permanent_uuid: "56dec7a4e0b8482aa22edc244ee37c51" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43517 } }
I20260620 02:52:51.763972 7850 raft_consensus.cc:348] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33858113a83d4f2da92c8f2364a93c83" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43733 } } peers { permanent_uuid: "66f25b3d96fa40f294be341e1266df2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36091 } } peers { permanent_uuid: "56dec7a4e0b8482aa22edc244ee37c51" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43517 } }
I20260620 02:52:51.764328 7850 raft_consensus.cc:374] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:51.764415 7850 raft_consensus.cc:729] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f25b3d96fa40f294be341e1266df2d, State: Initialized, Role: FOLLOWER
I20260620 02:52:51.765005 7850 consensus_queue.cc:260] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33858113a83d4f2da92c8f2364a93c83" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43733 } } peers { permanent_uuid: "66f25b3d96fa40f294be341e1266df2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36091 } } peers { permanent_uuid: "56dec7a4e0b8482aa22edc244ee37c51" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43517 } }
I20260620 02:52:51.767344 7847 ts_tablet_manager.cc:1428] T 34fd838245f34dd586f2e5772fa3c840 P 33858113a83d4f2da92c8f2364a93c83: Time spent starting tablet: real 0.022s user 0.015s sys 0.000s
I20260620 02:52:51.767925 7849 raft_consensus.cc:348] T 34fd838245f34dd586f2e5772fa3c840 P 56dec7a4e0b8482aa22edc244ee37c51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33858113a83d4f2da92c8f2364a93c83" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43733 } } peers { permanent_uuid: "66f25b3d96fa40f294be341e1266df2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36091 } } peers { permanent_uuid: "56dec7a4e0b8482aa22edc244ee37c51" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43517 } }
I20260620 02:52:51.768350 7849 raft_consensus.cc:374] T 34fd838245f34dd586f2e5772fa3c840 P 56dec7a4e0b8482aa22edc244ee37c51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:51.768445 7849 raft_consensus.cc:729] T 34fd838245f34dd586f2e5772fa3c840 P 56dec7a4e0b8482aa22edc244ee37c51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56dec7a4e0b8482aa22edc244ee37c51, State: Initialized, Role: FOLLOWER
I20260620 02:52:51.769034 7849 consensus_queue.cc:260] T 34fd838245f34dd586f2e5772fa3c840 P 56dec7a4e0b8482aa22edc244ee37c51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33858113a83d4f2da92c8f2364a93c83" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43733 } } peers { permanent_uuid: "66f25b3d96fa40f294be341e1266df2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36091 } } peers { permanent_uuid: "56dec7a4e0b8482aa22edc244ee37c51" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43517 } }
I20260620 02:52:51.772635 7849 ts_tablet_manager.cc:1428] T 34fd838245f34dd586f2e5772fa3c840 P 56dec7a4e0b8482aa22edc244ee37c51: Time spent starting tablet: real 0.015s user 0.007s sys 0.009s
I20260620 02:52:51.773013 7815 heartbeater.cc:503] Master 127.0.97.62:45057 was elected leader, sending a full tablet report...
I20260620 02:52:51.774061 7674 heartbeater.cc:503] Master 127.0.97.62:45057 was elected leader, sending a full tablet report...
I20260620 02:52:51.777325 7850 ts_tablet_manager.cc:1428] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d: Time spent starting tablet: real 0.023s user 0.011s sys 0.007s
W20260620 02:52:51.814440 7528 tablet.cc:2369] T 34fd838245f34dd586f2e5772fa3c840 P 33858113a83d4f2da92c8f2364a93c83: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:52:51.835366 7817 tablet.cc:2369] T 34fd838245f34dd586f2e5772fa3c840 P 56dec7a4e0b8482aa22edc244ee37c51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:52:51.907675 7854 raft_consensus.cc:482] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:52:51.907946 7854 raft_consensus.cc:504] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33858113a83d4f2da92c8f2364a93c83" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43733 } } peers { permanent_uuid: "66f25b3d96fa40f294be341e1266df2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36091 } } peers { permanent_uuid: "56dec7a4e0b8482aa22edc244ee37c51" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43517 } }
I20260620 02:52:51.909722 7854 leader_election.cc:288] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33858113a83d4f2da92c8f2364a93c83 (127.0.97.1:43733), 56dec7a4e0b8482aa22edc244ee37c51 (127.0.97.3:43517)
I20260620 02:52:51.918758 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:51.910351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40699 (local address 127.0.97.3:43517)
0620 02:52:51.910664 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:51.910673 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:51.911592 (+ 919us) server_negotiation.cc:407] Connection header received
0620 02:52:51.911712 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:51.911718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:51.911792 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:51.911883 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:51.912895 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.913992 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.916543 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.917765 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.917902 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:51.918010 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:51.918329 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 02:52:51.918503 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260620 02:52:51.918979 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:51.909994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43517 (local address 127.0.97.2:40699)
0620 02:52:51.911485 (+ 1491us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:51.911501 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:51.911605 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:51.912040 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:51.912045 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:51.912174 (+ 129us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:51.912585 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:51.912596 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.914206 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.914210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:51.916347 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:51.916360 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.917966 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.917974 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:51.918142 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:51.918184 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 02:52:51.918281 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 02:52:51.918446 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1391,"mutex_wait_us":1,"spinlock_wait_cycles":177920,"thread_start_us":1232,"threads_started":1}
I20260620 02:52:51.919661 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:51.910150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45767 (local address 127.0.97.1:43733)
0620 02:52:51.910454 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:51.910463 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:51.911350 (+ 887us) server_negotiation.cc:407] Connection header received
0620 02:52:51.911693 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:51.911700 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:51.911783 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:51.911910 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:51.912733 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.913992 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.917081 (+ 3089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.918229 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.918398 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:51.918495 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:51.919235 (+ 740us) server_negotiation.cc:299] Negotiation successful
0620 02:52:51.919365 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:52:51.920195 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:51.909936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43733 (local address 127.0.97.2:45767)
0620 02:52:51.911216 (+ 1280us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:51.911239 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:51.911485 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:51.911944 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:51.911948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:51.911978 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:51.912553 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:51.912563 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.914268 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.914273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:51.916934 (+ 2661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:51.916948 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.918659 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.918665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:51.918845 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:51.919077 (+ 232us) client_negotiation.cc:769] Sending connection context
0620 02:52:51.919854 (+ 777us) client_negotiation.cc:240] Negotiation successful
0620 02:52:51.919977 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1130,"thread_start_us":945,"threads_started":1}
I20260620 02:52:51.920137 7769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34fd838245f34dd586f2e5772fa3c840" candidate_uuid: "66f25b3d96fa40f294be341e1266df2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56dec7a4e0b8482aa22edc244ee37c51" is_pre_election: true
I20260620 02:52:51.920771 7769 raft_consensus.cc:2454] T 34fd838245f34dd586f2e5772fa3c840 P 56dec7a4e0b8482aa22edc244ee37c51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66f25b3d96fa40f294be341e1266df2d in term 0.
I20260620 02:52:51.921254 7480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34fd838245f34dd586f2e5772fa3c840" candidate_uuid: "66f25b3d96fa40f294be341e1266df2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33858113a83d4f2da92c8f2364a93c83" is_pre_election: true
I20260620 02:52:51.921501 7562 leader_election.cc:302] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56dec7a4e0b8482aa22edc244ee37c51, 66f25b3d96fa40f294be341e1266df2d; no voters:
I20260620 02:52:51.921767 7480 raft_consensus.cc:2454] T 34fd838245f34dd586f2e5772fa3c840 P 33858113a83d4f2da92c8f2364a93c83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66f25b3d96fa40f294be341e1266df2d in term 0.
I20260620 02:52:51.921932 7854 raft_consensus.cc:2781] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:52:51.922072 7854 raft_consensus.cc:482] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:52:51.922174 7854 raft_consensus.cc:3037] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:51.925141 7854 raft_consensus.cc:504] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33858113a83d4f2da92c8f2364a93c83" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43733 } } peers { permanent_uuid: "66f25b3d96fa40f294be341e1266df2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36091 } } peers { permanent_uuid: "56dec7a4e0b8482aa22edc244ee37c51" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43517 } }
I20260620 02:52:51.925669 7854 leader_election.cc:288] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d [CANDIDATE]: Term 1 election: Requested vote from peers 33858113a83d4f2da92c8f2364a93c83 (127.0.97.1:43733), 56dec7a4e0b8482aa22edc244ee37c51 (127.0.97.3:43517)
I20260620 02:52:51.926222 7480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34fd838245f34dd586f2e5772fa3c840" candidate_uuid: "66f25b3d96fa40f294be341e1266df2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33858113a83d4f2da92c8f2364a93c83"
I20260620 02:52:51.926208 7769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34fd838245f34dd586f2e5772fa3c840" candidate_uuid: "66f25b3d96fa40f294be341e1266df2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56dec7a4e0b8482aa22edc244ee37c51"
I20260620 02:52:51.926450 7480 raft_consensus.cc:3037] T 34fd838245f34dd586f2e5772fa3c840 P 33858113a83d4f2da92c8f2364a93c83 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:51.926451 7769 raft_consensus.cc:3037] T 34fd838245f34dd586f2e5772fa3c840 P 56dec7a4e0b8482aa22edc244ee37c51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:51.930032 7769 raft_consensus.cc:2454] T 34fd838245f34dd586f2e5772fa3c840 P 56dec7a4e0b8482aa22edc244ee37c51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66f25b3d96fa40f294be341e1266df2d in term 1.
I20260620 02:52:51.930032 7480 raft_consensus.cc:2454] T 34fd838245f34dd586f2e5772fa3c840 P 33858113a83d4f2da92c8f2364a93c83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66f25b3d96fa40f294be341e1266df2d in term 1.
I20260620 02:52:51.930493 7561 leader_election.cc:302] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 33858113a83d4f2da92c8f2364a93c83, 56dec7a4e0b8482aa22edc244ee37c51, 66f25b3d96fa40f294be341e1266df2d; no voters:
I20260620 02:52:51.930886 7854 raft_consensus.cc:2781] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:51.931377 7854 raft_consensus.cc:686] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d [term 1 LEADER]: Becoming Leader. State: Replica: 66f25b3d96fa40f294be341e1266df2d, State: Running, Role: LEADER
I20260620 02:52:51.931912 7854 consensus_queue.cc:237] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33858113a83d4f2da92c8f2364a93c83" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43733 } } peers { permanent_uuid: "66f25b3d96fa40f294be341e1266df2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36091 } } peers { permanent_uuid: "56dec7a4e0b8482aa22edc244ee37c51" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43517 } }
W20260620 02:52:51.935130 7676 tablet.cc:2369] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:52:51.937335 6555 catalog_manager.cc:5495] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 66f25b3d96fa40f294be341e1266df2d (127.0.97.2). New cstate: current_term: 1 leader_uuid: "66f25b3d96fa40f294be341e1266df2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33858113a83d4f2da92c8f2364a93c83" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66f25b3d96fa40f294be341e1266df2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56dec7a4e0b8482aa22edc244ee37c51" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43517 } health_report { overall_health: UNKNOWN } } }
I20260620 02:52:52.001478 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:51.995710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36068 (local address 127.0.97.2:36091)
0620 02:52:51.995918 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:51.995925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:51.995947 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:52:51.996160 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:51.996165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:51.996247 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:51.996426 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:52:51.996841 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.997633 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.999156 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:51.999601 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:51.999752 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:52.000270 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:52:52.000808 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:52:52.001005 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:52.001092 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 02:52:52.001182 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 02:52:52.009933 7480 raft_consensus.cc:1261] T 34fd838245f34dd586f2e5772fa3c840 P 33858113a83d4f2da92c8f2364a93c83 [term 1 FOLLOWER]: Refusing update from remote peer 66f25b3d96fa40f294be341e1266df2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:52.009936 7769 raft_consensus.cc:1261] T 34fd838245f34dd586f2e5772fa3c840 P 56dec7a4e0b8482aa22edc244ee37c51 [term 1 FOLLOWER]: Refusing update from remote peer 66f25b3d96fa40f294be341e1266df2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:52.011013 7858 consensus_queue.cc:1035] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "33858113a83d4f2da92c8f2364a93c83" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260620 02:52:52.011795 7854 consensus_queue.cc:1035] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "56dec7a4e0b8482aa22edc244ee37c51" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:52:52.043569 6556 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:52:52.043886 6556 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:52:52.051100 6556 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1caa9396c123467c8288bc9f054a1c1f: Sending DeleteTablet for 3 replicas of tablet 34fd838245f34dd586f2e5772fa3c840
I20260620 02:52:52.052400 7608 tablet_service.cc:1514] Processing DeleteTablet for tablet 34fd838245f34dd586f2e5772fa3c840 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:52 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36066
I20260620 02:52:52.052449 7749 tablet_service.cc:1514] Processing DeleteTablet for tablet 34fd838245f34dd586f2e5772fa3c840 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:52 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54914
I20260620 02:52:52.052448 7460 tablet_service.cc:1514] Processing DeleteTablet for tablet 34fd838245f34dd586f2e5772fa3c840 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:52 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49936
I20260620 02:52:52.053299 7871 tablet_replica.cc:330] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d: stopping tablet replica
I20260620 02:52:52.053767 7871 raft_consensus.cc:2229] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:52:52.054476 7871 raft_consensus.cc:2258] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:52.057245 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 6524
I20260620 02:52:52.058328 6586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:52.058718 7871 ts_tablet_manager.cc:1905] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:52.072983 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:52.055792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59906 (local address 127.0.97.61:41347)
0620 02:52:52.055944 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:52.055950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:52.055967 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:52:52.056198 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:52.056204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:52.056294 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:52.056366 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:52.059011 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:52.060114 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:52.061965 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:52.062653 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:52.062752 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:52.066208 (+ 3456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:52.066215 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:52.066219 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:52.066258 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:52.069808 (+ 3550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:52.070658 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:52.070665 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:52.070669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:52.070803 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:52.071250 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:52.071255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:52.071259 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:52.071600 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:52.071808 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:52.072403 (+ 595us) server_negotiation.cc:299] Negotiation successful
0620 02:52:52.072628 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 02:52:52.073971 7873 tablet_replica.cc:330] T 34fd838245f34dd586f2e5772fa3c840 P 56dec7a4e0b8482aa22edc244ee37c51: stopping tablet replica
I20260620 02:52:52.075865 7873 raft_consensus.cc:2229] T 34fd838245f34dd586f2e5772fa3c840 P 56dec7a4e0b8482aa22edc244ee37c51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:52:52.077032 7872 tablet_replica.cc:330] T 34fd838245f34dd586f2e5772fa3c840 P 33858113a83d4f2da92c8f2364a93c83: stopping tablet replica
I20260620 02:52:52.080060 7873 raft_consensus.cc:2258] T 34fd838245f34dd586f2e5772fa3c840 P 56dec7a4e0b8482aa22edc244ee37c51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:52.082582 7873 ts_tablet_manager.cc:1905] T 34fd838245f34dd586f2e5772fa3c840 P 56dec7a4e0b8482aa22edc244ee37c51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:52.085495 7872 raft_consensus.cc:2229] T 34fd838245f34dd586f2e5772fa3c840 P 33858113a83d4f2da92c8f2364a93c83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:52:52.086082 7872 raft_consensus.cc:2258] T 34fd838245f34dd586f2e5772fa3c840 P 33858113a83d4f2da92c8f2364a93c83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:52.087316 7871 ts_tablet_manager.cc:1918] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:52:52.087513 7871 log.cc:1196] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-1/wal/wals/34fd838245f34dd586f2e5772fa3c840
I20260620 02:52:52.088171 7871 ts_tablet_manager.cc:1939] T 34fd838245f34dd586f2e5772fa3c840 P 66f25b3d96fa40f294be341e1266df2d: Deleting consensus metadata
I20260620 02:52:52.090998 7872 ts_tablet_manager.cc:1905] T 34fd838245f34dd586f2e5772fa3c840 P 33858113a83d4f2da92c8f2364a93c83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:52.093997 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:52.056647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54764 (local address 127.0.97.60:43075)
0620 02:52:52.056813 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:52.056818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:52.074778 (+ 17960us) server_negotiation.cc:407] Connection header received
0620 02:52:52.075475 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:52.075481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:52.075567 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:52.075639 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:52.077875 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:52.079041 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:52.081734 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:52.082426 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:52.082528 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:52.086153 (+ 3625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:52.086160 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:52.086165 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:52.086207 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:52.090589 (+ 4382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:52.091556 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:52.091562 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:52.091566 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:52.091720 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:52.092180 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:52.092186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:52.092189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:52.092544 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:52.092769 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:52.093355 (+ 586us) server_negotiation.cc:299] Negotiation successful
0620 02:52:52.093576 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 02:52:52.098536 7873 ts_tablet_manager.cc:1918] T 34fd838245f34dd586f2e5772fa3c840 P 56dec7a4e0b8482aa22edc244ee37c51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:52:52.098740 7873 log.cc:1196] T 34fd838245f34dd586f2e5772fa3c840 P 56dec7a4e0b8482aa22edc244ee37c51: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-2/wal/wals/34fd838245f34dd586f2e5772fa3c840
I20260620 02:52:52.099299 7873 ts_tablet_manager.cc:1939] T 34fd838245f34dd586f2e5772fa3c840 P 56dec7a4e0b8482aa22edc244ee37c51: Deleting consensus metadata
I20260620 02:52:52.103744 7872 ts_tablet_manager.cc:1918] T 34fd838245f34dd586f2e5772fa3c840 P 33858113a83d4f2da92c8f2364a93c83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:52:52.103911 7872 log.cc:1196] T 34fd838245f34dd586f2e5772fa3c840 P 33858113a83d4f2da92c8f2364a93c83: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-0/wal/wals/34fd838245f34dd586f2e5772fa3c840
I20260620 02:52:52.104468 7872 ts_tablet_manager.cc:1939] T 34fd838245f34dd586f2e5772fa3c840 P 33858113a83d4f2da92c8f2364a93c83: Deleting consensus metadata
I20260620 02:52:52.213323 6542 catalog_manager.cc:4841] TS 56dec7a4e0b8482aa22edc244ee37c51 (127.0.97.3:43517): tablet 34fd838245f34dd586f2e5772fa3c840 (table test-table [id=f0a9678eaeaf42a4a3ec219b873ccd79]) successfully deleted
I20260620 02:52:52.214999 6541 catalog_manager.cc:4841] TS 33858113a83d4f2da92c8f2364a93c83 (127.0.97.1:43733): tablet 34fd838245f34dd586f2e5772fa3c840 (table test-table [id=f0a9678eaeaf42a4a3ec219b873ccd79]) successfully deleted
I20260620 02:52:52.215327 6541 catalog_manager.cc:4841] TS 66f25b3d96fa40f294be341e1266df2d (127.0.97.2:36091): tablet 34fd838245f34dd586f2e5772fa3c840 (table test-table [id=f0a9678eaeaf42a4a3ec219b873ccd79]) successfully deleted
I20260620 02:52:52.215852 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 6524
I20260620 02:52:52.264829 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:52.250221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46964 (local address 127.0.97.60:43075)
0620 02:52:52.250590 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:52.250598 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:52.251546 (+ 948us) server_negotiation.cc:407] Connection header received
0620 02:52:52.251810 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:52.251816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:52.251907 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:52.251982 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:52.253641 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:52.254358 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:52.255645 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:52.256061 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:52.256194 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:52.259467 (+ 3273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:52.259471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:52.259475 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:52.259508 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:52.262075 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:52.262759 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:52.262767 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:52.262770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:52.262903 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:52.263302 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:52.263310 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:52.263314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:52.263593 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:52.263767 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:52.264216 (+ 449us) server_negotiation.cc:299] Negotiation successful
0620 02:52:52.264414 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 02:52:52.267267 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:52.251738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49912 (local address 127.0.97.61:41347)
0620 02:52:52.251904 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:52.251909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:52.252014 (+ 105us) server_negotiation.cc:407] Connection header received
0620 02:52:52.252169 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:52.252174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:52.252245 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:52.252351 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:52.254298 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:52.255438 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:52.257369 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:52.258127 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:52.258286 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:52.261325 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:52.261331 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:52.261336 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:52.261373 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:52.264662 (+ 3289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:52.265296 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:52.265303 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:52.265307 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:52.265441 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:52.265805 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:52.265810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:52.265812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:52.266155 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:52.266414 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:52.266773 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 02:52:52.266966 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 02:52:53.032011 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.031279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.97.2:44055)
0620 02:52:53.031734 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:53.031843 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":260,"threads_started":1}
W20260620 02:52:53.032452 7674 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45057 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111)
I20260620 02:52:53.039701 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.038574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.97.3:45151)
0620 02:52:53.039304 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:53.039474 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":465,"threads_started":1}
I20260620 02:52:53.040150 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.038943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.97.1:36689)
0620 02:52:53.039598 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:53.039724 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":421,"threads_started":1}
W20260620 02:52:53.040170 7815 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45057 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111)
W20260620 02:52:53.040712 7526 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45057 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111)
I20260620 02:52:53.551671 7886 raft_consensus.cc:482] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1caa9396c123467c8288bc9f054a1c1f)
I20260620 02:52:53.551970 7886 raft_consensus.cc:504] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: false } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: false } }
I20260620 02:52:53.552393 7885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1caa9396c123467c8288bc9f054a1c1f)
I20260620 02:52:53.552625 7885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: false } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: false } }
I20260620 02:52:53.553439 7886 leader_election.cc:288] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1caa9396c123467c8288bc9f054a1c1f (127.0.97.62:45057), 8d11130cfec9434094b0340a81f4230e (127.0.97.60:43075)
I20260620 02:52:53.553987 7885 leader_election.cc:288] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1caa9396c123467c8288bc9f054a1c1f (127.0.97.62:45057), 982c8b2c204d443087f6c9f667437974 (127.0.97.61:41347)
I20260620 02:52:53.554884 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.553681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.0.1:39262)
0620 02:52:53.554489 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:53.554630 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":639,"threads_started":1}
W20260620 02:52:53.555208 7141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111)
I20260620 02:52:53.555007 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.554172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.0.1:39264)
0620 02:52:53.554650 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:53.554752 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":321,"threads_started":1}
I20260620 02:52:53.556991 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.556602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.0.1:39266)
0620 02:52:53.556732 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:53.556847 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 02:52:53.557140 7141 leader_election.cc:334] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1caa9396c123467c8288bc9f054a1c1f (127.0.97.62:45057): Network error: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111)
W20260620 02:52:53.557458 7331 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111)
I20260620 02:52:53.564200 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.553640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43075 (local address 127.0.0.1:46968)
0620 02:52:53.554069 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:53.554087 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:53.554203 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:53.557859 (+ 3656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:53.557864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:53.557903 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:53.558507 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:53.558523 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:53.559933 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:53.559939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:53.561201 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:53.561212 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:53.563488 (+ 2276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:53.563493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:53.563602 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:53.563620 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 02:52:53.563721 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 02:52:53.563847 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":376,"threads_started":1}
I20260620 02:52:53.564229 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.555772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46968 (local address 127.0.97.60:43075)
0620 02:52:53.557470 (+ 1698us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:53.557474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:53.557491 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:52:53.557564 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:53.557570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:53.557722 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:53.557836 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:53.558712 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:53.559800 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:53.561357 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:53.562315 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:53.562491 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:53.562598 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:53.563745 (+ 1147us) server_negotiation.cc:299] Negotiation successful
0620 02:52:53.563867 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":1344,"thread_start_us":1124,"threads_started":1}
I20260620 02:52:53.564649 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.555102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49924 (local address 127.0.97.61:41347)
0620 02:52:53.555881 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:53.555888 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:53.555909 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:52:53.555977 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:53.555982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:53.556075 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:53.556187 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:53.557817 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:53.559129 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:53.562462 (+ 3333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:53.563263 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:53.563362 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:53.563422 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:53.563953 (+ 531us) server_negotiation.cc:299] Negotiation successful
0620 02:52:53.564144 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":317,"threads_started":1}
I20260620 02:52:53.565338 7356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "982c8b2c204d443087f6c9f667437974" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8d11130cfec9434094b0340a81f4230e" is_pre_election: true
I20260620 02:52:53.565421 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.554831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41347 (local address 127.0.0.1:49924)
0620 02:52:53.555337 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:53.555360 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:53.555483 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:53.556253 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:53.556261 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:53.556301 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:53.557641 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:53.557658 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:53.559292 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:53.559300 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:53.562267 (+ 2967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:53.562286 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:53.563460 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:53.563468 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:53.563722 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:53.563775 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 02:52:53.564237 (+ 462us) client_negotiation.cc:240] Negotiation successful
0620 02:52:53.564992 (+ 755us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":28,"thread_start_us":734,"threads_started":1}
I20260620 02:52:53.565683 7356 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982c8b2c204d443087f6c9f667437974 in term 1.
I20260620 02:52:53.565842 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.565504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.0.1:39274)
0620 02:52:53.565655 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:53.565719 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 02:52:53.566118 7331 leader_election.cc:334] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1caa9396c123467c8288bc9f054a1c1f (127.0.97.62:45057): Network error: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111)
I20260620 02:52:53.566627 7142 leader_election.cc:302] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d11130cfec9434094b0340a81f4230e, 982c8b2c204d443087f6c9f667437974; no voters: 1caa9396c123467c8288bc9f054a1c1f
I20260620 02:52:53.566875 7166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d11130cfec9434094b0340a81f4230e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "982c8b2c204d443087f6c9f667437974" is_pre_election: true
I20260620 02:52:53.567059 7886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 02:52:53.567178 7886 raft_consensus.cc:482] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1caa9396c123467c8288bc9f054a1c1f)
I20260620 02:52:53.567276 7886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:52:53.569470 7886 raft_consensus.cc:504] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: false } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: false } }
I20260620 02:52:53.569865 7166 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8d11130cfec9434094b0340a81f4230e in current term 2: Already voted for candidate 982c8b2c204d443087f6c9f667437974 in this term.
I20260620 02:52:53.570557 7886 leader_election.cc:288] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [CANDIDATE]: Term 2 election: Requested vote from peers 1caa9396c123467c8288bc9f054a1c1f (127.0.97.62:45057), 8d11130cfec9434094b0340a81f4230e (127.0.97.60:43075)
I20260620 02:52:53.570744 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.570196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.0.1:39290)
0620 02:52:53.570443 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:53.570525 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 02:52:53.571285 7331 leader_election.cc:302] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d11130cfec9434094b0340a81f4230e; no voters: 1caa9396c123467c8288bc9f054a1c1f, 982c8b2c204d443087f6c9f667437974
I20260620 02:52:53.571383 7356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "982c8b2c204d443087f6c9f667437974" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8d11130cfec9434094b0340a81f4230e"
I20260620 02:52:53.571609 7356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:52:53.571765 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.571375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.0.1:39292)
0620 02:52:53.571532 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:53.571633 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 02:52:53.571959 7141 leader_election.cc:334] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1caa9396c123467c8288bc9f054a1c1f (127.0.97.62:45057): Network error: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111)
I20260620 02:52:53.573884 7356 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982c8b2c204d443087f6c9f667437974 in term 2.
I20260620 02:52:53.574095 7885 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 02:52:53.574348 7142 leader_election.cc:302] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d11130cfec9434094b0340a81f4230e, 982c8b2c204d443087f6c9f667437974; no voters: 1caa9396c123467c8288bc9f054a1c1f
I20260620 02:52:53.574703 7886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 02:52:53.575187 7886 raft_consensus.cc:686] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [term 2 LEADER]: Becoming Leader. State: Replica: 982c8b2c204d443087f6c9f667437974, State: Running, Role: LEADER
I20260620 02:52:53.575645 7886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: false } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: false } }
I20260620 02:52:53.577760 7895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 982c8b2c204d443087f6c9f667437974. Latest consensus state: current_term: 2 leader_uuid: "982c8b2c204d443087f6c9f667437974" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: false } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: false } } }
I20260620 02:52:53.578195 7895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:53.578882 7897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:52:53.583191 7897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:52:53.584051 7897 catalog_manager.cc:1259] Loaded cluster ID: 13341c1efd294e0597c84690d18aca07
I20260620 02:52:53.584128 7897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:52:53.585378 7897 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:52:53.586431 7897 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974: Loaded TSK: 0
I20260620 02:52:53.587438 7897 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:52:53.609120 7155 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:52:53.616256 7155 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:52:53.620306 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.619918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.0.1:39310)
0620 02:52:53.620083 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:53.620167 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 02:52:53.620357 7356 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [term 2 FOLLOWER]: Refusing update from remote peer 982c8b2c204d443087f6c9f667437974: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 02:52:53.621124 7895 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 02:52:53.621390 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.620979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.0.1:39314)
0620 02:52:53.621161 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:53.621246 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 02:52:53.621621 7141 consensus_peers.cc:487] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 -> Peer 1caa9396c123467c8288bc9f054a1c1f (127.0.97.62:45057): Couldn't send request to peer 1caa9396c123467c8288bc9f054a1c1f. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 02:52:53.624073 7885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 982c8b2c204d443087f6c9f667437974. Latest consensus state: current_term: 2 leader_uuid: "982c8b2c204d443087f6c9f667437974" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: false } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: false } } }
I20260620 02:52:53.624365 7885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:52:53.625537 7898 mvcc.cc:205] Tried to move back new op lower bound from 7298760595942559744 to 7298760595769839616. Current Snapshot: MvccSnapshot[applied={T|T < 7298760595942559744}]
I20260620 02:52:53.625919 7886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "982c8b2c204d443087f6c9f667437974" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: false } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: false } } }
I20260620 02:52:53.626220 7886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974 [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:53.626817 7885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "982c8b2c204d443087f6c9f667437974" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1caa9396c123467c8288bc9f054a1c1f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45057 } } peers { permanent_uuid: "982c8b2c204d443087f6c9f667437974" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41347 } attrs { promote: false } } peers { permanent_uuid: "8d11130cfec9434094b0340a81f4230e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43075 } attrs { promote: false } } }
I20260620 02:52:53.627063 7885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d11130cfec9434094b0340a81f4230e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:52:53.627410 7899 mvcc.cc:205] Tried to move back new op lower bound from 7298760595942559744 to 7298760595769839616. Current Snapshot: MvccSnapshot[applied={T|T < 7298760595942559744}]
I20260620 02:52:53.643278 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.634410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43517 (local address 127.0.0.1:52148)
0620 02:52:53.634563 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:53.634582 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:53.634672 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:53.635273 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:53.635278 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:53.635303 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:53.635492 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:53.635501 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:53.637047 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:53.637053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:53.639907 (+ 2854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:53.639927 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:53.642181 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:53.642190 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:53.642477 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:53.642536 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 02:52:53.642616 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 02:52:53.642811 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 02:52:53.644587 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.634559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52148 (local address 127.0.97.3:43517)
0620 02:52:53.635001 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:53.635005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:53.635019 (+ 14us) server_negotiation.cc:407] Connection header received
0620 02:52:53.635088 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:53.635092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:53.635150 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:53.635259 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:53.635624 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:53.636853 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:53.640158 (+ 3305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:53.641965 (+ 1807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:53.643688 (+ 1723us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:53.643847 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:53.643923 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 02:52:53.644102 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":263,"threads_started":1}
I20260620 02:52:53.645358 7749 tablet_service.cc:1467] Processing CreateTablet for tablet 2328b5e204644b5a95f1828ca1770609 (DEFAULT_TABLE table=test-table [id=0f39928cb15d4bf69f183fae86b9a734]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:53.645857 7749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2328b5e204644b5a95f1828ca1770609. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:53.651640 7908 tablet_bootstrap.cc:492] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51: Bootstrap starting.
I20260620 02:52:53.653435 7908 tablet_bootstrap.cc:654] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:53.655774 7908 tablet_bootstrap.cc:492] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51: No bootstrap required, opened a new log
I20260620 02:52:53.655926 7908 ts_tablet_manager.cc:1397] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 02:52:53.656742 7908 raft_consensus.cc:348] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56dec7a4e0b8482aa22edc244ee37c51" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43517 } } peers { permanent_uuid: "66f25b3d96fa40f294be341e1266df2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36091 } } peers { permanent_uuid: "33858113a83d4f2da92c8f2364a93c83" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43733 } }
I20260620 02:52:53.656996 7908 raft_consensus.cc:374] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:53.657084 7908 raft_consensus.cc:729] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56dec7a4e0b8482aa22edc244ee37c51, State: Initialized, Role: FOLLOWER
I20260620 02:52:53.657408 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.635338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36091 (local address 127.0.0.1:44268)
0620 02:52:53.635549 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:53.635576 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:53.635700 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:53.650678 (+ 14978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:53.650683 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:53.650707 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:53.651081 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:53.651095 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:53.652626 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:53.652635 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:53.654691 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:53.654710 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:53.656234 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:53.656242 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:53.656789 (+ 547us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:53.656841 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 02:52:53.656937 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 02:52:53.657086 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":29}
I20260620 02:52:53.657348 7908 consensus_queue.cc:260] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56dec7a4e0b8482aa22edc244ee37c51" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43517 } } peers { permanent_uuid: "66f25b3d96fa40f294be341e1266df2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36091 } } peers { permanent_uuid: "33858113a83d4f2da92c8f2364a93c83" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43733 } }
I20260620 02:52:53.658308 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.635749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44268 (local address 127.0.97.2:36091)
0620 02:52:53.650307 (+ 14558us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:53.650313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:53.650333 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:52:53.650429 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:53.650435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:53.650514 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:53.651243 (+ 729us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:53.651431 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:53.652449 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:53.654869 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:53.655821 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:53.655961 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:53.656069 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:53.657776 (+ 1707us) server_negotiation.cc:299] Negotiation successful
0620 02:52:53.657955 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14420,"thread_start_us":14323,"threads_started":1}
I20260620 02:52:53.658311 7908 ts_tablet_manager.cc:1428] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 02:52:53.659183 7608 tablet_service.cc:1467] Processing CreateTablet for tablet 2328b5e204644b5a95f1828ca1770609 (DEFAULT_TABLE table=test-table [id=0f39928cb15d4bf69f183fae86b9a734]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:53.659754 7608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2328b5e204644b5a95f1828ca1770609. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:53.660039 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.635643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43733 (local address 127.0.0.1:47854)
0620 02:52:53.636200 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:53.636219 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:53.636325 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:53.653252 (+ 16927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:53.653258 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:53.653289 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:53.653731 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:53.653743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:53.655130 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:53.655139 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:53.657409 (+ 2270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:53.657423 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:53.659277 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:53.659285 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:53.659476 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:53.659517 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 02:52:53.659577 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 02:52:53.659674 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":32,"thread_start_us":506,"threads_started":1}
I20260620 02:52:53.660868 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.636074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47854 (local address 127.0.97.1:43733)
0620 02:52:53.652937 (+ 16863us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:53.652943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:53.652961 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:52:53.653027 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:53.653031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:53.653105 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:53.653212 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:53.653949 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:53.654943 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:53.657611 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:53.659033 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:53.660408 (+ 1375us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:53.660497 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:53.660541 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 02:52:53.660638 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16733,"thread_start_us":16637,"threads_started":1}
I20260620 02:52:53.661641 7460 tablet_service.cc:1467] Processing CreateTablet for tablet 2328b5e204644b5a95f1828ca1770609 (DEFAULT_TABLE table=test-table [id=0f39928cb15d4bf69f183fae86b9a734]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:53.662076 7460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2328b5e204644b5a95f1828ca1770609. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:53.663307 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.659040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.97.3:60393)
0620 02:52:53.662898 (+ 3858us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:53.663041 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3703,"thread_start_us":3543,"threads_started":1}
I20260620 02:52:53.668464 7912 tablet_bootstrap.cc:492] T 2328b5e204644b5a95f1828ca1770609 P 33858113a83d4f2da92c8f2364a93c83: Bootstrap starting.
I20260620 02:52:53.668651 7913 tablet_bootstrap.cc:492] T 2328b5e204644b5a95f1828ca1770609 P 66f25b3d96fa40f294be341e1266df2d: Bootstrap starting.
I20260620 02:52:53.669006 7816 heartbeater.cc:503] Master 127.0.97.61:41347 was elected leader, sending a full tablet report...
I20260620 02:52:53.669184 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.668726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.97.3:37579)
0620 02:52:53.668904 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:53.669017 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 02:52:53.670301 7913 tablet_bootstrap.cc:654] T 2328b5e204644b5a95f1828ca1770609 P 66f25b3d96fa40f294be341e1266df2d: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:53.671036 7912 tablet_bootstrap.cc:654] T 2328b5e204644b5a95f1828ca1770609 P 33858113a83d4f2da92c8f2364a93c83: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:53.673100 7912 tablet_bootstrap.cc:492] T 2328b5e204644b5a95f1828ca1770609 P 33858113a83d4f2da92c8f2364a93c83: No bootstrap required, opened a new log
I20260620 02:52:53.673273 7912 ts_tablet_manager.cc:1397] T 2328b5e204644b5a95f1828ca1770609 P 33858113a83d4f2da92c8f2364a93c83: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 02:52:53.673938 7912 raft_consensus.cc:348] T 2328b5e204644b5a95f1828ca1770609 P 33858113a83d4f2da92c8f2364a93c83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56dec7a4e0b8482aa22edc244ee37c51" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43517 } } peers { permanent_uuid: "66f25b3d96fa40f294be341e1266df2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36091 } } peers { permanent_uuid: "33858113a83d4f2da92c8f2364a93c83" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43733 } }
I20260620 02:52:53.674130 7912 raft_consensus.cc:374] T 2328b5e204644b5a95f1828ca1770609 P 33858113a83d4f2da92c8f2364a93c83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:53.674256 7912 raft_consensus.cc:729] T 2328b5e204644b5a95f1828ca1770609 P 33858113a83d4f2da92c8f2364a93c83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33858113a83d4f2da92c8f2364a93c83, State: Initialized, Role: FOLLOWER
I20260620 02:52:53.674566 7912 consensus_queue.cc:260] T 2328b5e204644b5a95f1828ca1770609 P 33858113a83d4f2da92c8f2364a93c83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56dec7a4e0b8482aa22edc244ee37c51" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43517 } } peers { permanent_uuid: "66f25b3d96fa40f294be341e1266df2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36091 } } peers { permanent_uuid: "33858113a83d4f2da92c8f2364a93c83" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43733 } }
I20260620 02:52:53.676672 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.675611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.97.1:36533)
0620 02:52:53.676226 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:53.676377 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":309,"threads_started":1}
I20260620 02:52:53.678061 7912 ts_tablet_manager.cc:1428] T 2328b5e204644b5a95f1828ca1770609 P 33858113a83d4f2da92c8f2364a93c83: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260620 02:52:53.678328 7527 heartbeater.cc:503] Master 127.0.97.61:41347 was elected leader, sending a full tablet report...
I20260620 02:52:53.679088 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.678629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.97.1:50749)
0620 02:52:53.678843 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:53.678930 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260620 02:52:53.679734 7913 tablet_bootstrap.cc:492] T 2328b5e204644b5a95f1828ca1770609 P 66f25b3d96fa40f294be341e1266df2d: No bootstrap required, opened a new log
I20260620 02:52:53.679893 7913 ts_tablet_manager.cc:1397] T 2328b5e204644b5a95f1828ca1770609 P 66f25b3d96fa40f294be341e1266df2d: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260620 02:52:53.680555 7913 raft_consensus.cc:348] T 2328b5e204644b5a95f1828ca1770609 P 66f25b3d96fa40f294be341e1266df2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56dec7a4e0b8482aa22edc244ee37c51" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43517 } } peers { permanent_uuid: "66f25b3d96fa40f294be341e1266df2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36091 } } peers { permanent_uuid: "33858113a83d4f2da92c8f2364a93c83" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43733 } }
I20260620 02:52:53.680733 7913 raft_consensus.cc:374] T 2328b5e204644b5a95f1828ca1770609 P 66f25b3d96fa40f294be341e1266df2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:53.680792 7913 raft_consensus.cc:729] T 2328b5e204644b5a95f1828ca1770609 P 66f25b3d96fa40f294be341e1266df2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f25b3d96fa40f294be341e1266df2d, State: Initialized, Role: FOLLOWER
I20260620 02:52:53.681002 7913 consensus_queue.cc:260] T 2328b5e204644b5a95f1828ca1770609 P 66f25b3d96fa40f294be341e1266df2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56dec7a4e0b8482aa22edc244ee37c51" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43517 } } peers { permanent_uuid: "66f25b3d96fa40f294be341e1266df2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36091 } } peers { permanent_uuid: "33858113a83d4f2da92c8f2364a93c83" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43733 } }
I20260620 02:52:53.681586 7913 ts_tablet_manager.cc:1428] T 2328b5e204644b5a95f1828ca1770609 P 66f25b3d96fa40f294be341e1266df2d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 02:52:53.683048 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.681979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.97.2:39935)
0620 02:52:53.682604 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:53.682723 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":275,"threads_started":1}
I20260620 02:52:53.684255 7675 heartbeater.cc:503] Master 127.0.97.61:41347 was elected leader, sending a full tablet report...
I20260620 02:52:53.684906 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.683930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.97.2:34941)
0620 02:52:53.684671 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:53.684753 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":641}
I20260620 02:52:53.810489 7910 raft_consensus.cc:482] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:52:53.810802 7910 raft_consensus.cc:504] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56dec7a4e0b8482aa22edc244ee37c51" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43517 } } peers { permanent_uuid: "66f25b3d96fa40f294be341e1266df2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36091 } } peers { permanent_uuid: "33858113a83d4f2da92c8f2364a93c83" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43733 } }
I20260620 02:52:53.812553 7910 leader_election.cc:288] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66f25b3d96fa40f294be341e1266df2d (127.0.97.2:36091), 33858113a83d4f2da92c8f2364a93c83 (127.0.97.1:43733)
I20260620 02:52:53.820050 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.812926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36091 (local address 127.0.97.3:39579)
0620 02:52:53.813127 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:53.813151 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:53.813318 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:53.813971 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:53.813976 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:53.814017 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:53.814355 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:53.814413 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:53.815778 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:53.815785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:53.817783 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:53.817797 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:53.819287 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:53.819294 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:53.819462 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:53.819499 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 02:52:53.819610 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 02:52:53.819758 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":1}
I20260620 02:52:53.820096 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.813008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39579 (local address 127.0.97.2:36091)
0620 02:52:53.813228 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:53.813232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:53.813284 (+ 52us) server_negotiation.cc:407] Connection header received
0620 02:52:53.813532 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:53.813538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:53.813633 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:53.813762 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:53.814571 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:53.815631 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:53.817975 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:53.819113 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:53.819217 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:53.819296 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:53.819666 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 02:52:53.819812 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 02:52:53.820839 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.813626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56061 (local address 127.0.97.1:43733)
0620 02:52:53.813871 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:53.813879 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:53.814124 (+ 245us) server_negotiation.cc:407] Connection header received
0620 02:52:53.814275 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:53.814281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:53.814400 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:53.814517 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:53.815112 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:53.816227 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:53.818535 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:53.819556 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:53.819679 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:53.819774 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:53.820353 (+ 579us) server_negotiation.cc:299] Negotiation successful
0620 02:52:53.820522 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 02:52:53.821571 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.813343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43733 (local address 127.0.97.3:56061)
0620 02:52:53.814008 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:53.814023 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:53.814135 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:53.814537 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:53.814543 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:53.814568 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:53.814941 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:53.814954 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:53.816398 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:53.816402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:53.818338 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:53.818355 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:53.819759 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:53.819766 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:53.819976 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:53.820072 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 02:52:53.821086 (+ 1014us) client_negotiation.cc:240] Negotiation successful
0620 02:52:53.821245 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":400,"threads_started":1}
I20260620 02:52:53.822517 7480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2328b5e204644b5a95f1828ca1770609" candidate_uuid: "56dec7a4e0b8482aa22edc244ee37c51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33858113a83d4f2da92c8f2364a93c83" is_pre_election: true
I20260620 02:52:53.822815 7480 raft_consensus.cc:2454] T 2328b5e204644b5a95f1828ca1770609 P 33858113a83d4f2da92c8f2364a93c83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56dec7a4e0b8482aa22edc244ee37c51 in term 0.
I20260620 02:52:53.822806 7628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2328b5e204644b5a95f1828ca1770609" candidate_uuid: "56dec7a4e0b8482aa22edc244ee37c51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66f25b3d96fa40f294be341e1266df2d" is_pre_election: true
I20260620 02:52:53.823207 7918 raft_consensus.cc:482] T 2328b5e204644b5a95f1828ca1770609 P 66f25b3d96fa40f294be341e1266df2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:52:53.823529 7918 raft_consensus.cc:504] T 2328b5e204644b5a95f1828ca1770609 P 66f25b3d96fa40f294be341e1266df2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56dec7a4e0b8482aa22edc244ee37c51" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43517 } } peers { permanent_uuid: "66f25b3d96fa40f294be341e1266df2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36091 } } peers { permanent_uuid: "33858113a83d4f2da92c8f2364a93c83" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43733 } }
I20260620 02:52:53.823796 7703 leader_election.cc:302] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33858113a83d4f2da92c8f2364a93c83, 56dec7a4e0b8482aa22edc244ee37c51; no voters:
I20260620 02:52:53.824018 7628 raft_consensus.cc:2454] T 2328b5e204644b5a95f1828ca1770609 P 66f25b3d96fa40f294be341e1266df2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56dec7a4e0b8482aa22edc244ee37c51 in term 0.
I20260620 02:52:53.824862 7910 raft_consensus.cc:2781] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:52:53.824999 7910 raft_consensus.cc:482] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:52:53.825099 7910 raft_consensus.cc:3037] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:53.825961 7918 leader_election.cc:288] T 2328b5e204644b5a95f1828ca1770609 P 66f25b3d96fa40f294be341e1266df2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56dec7a4e0b8482aa22edc244ee37c51 (127.0.97.3:43517), 33858113a83d4f2da92c8f2364a93c83 (127.0.97.1:43733)
I20260620 02:52:53.826000 7769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2328b5e204644b5a95f1828ca1770609" candidate_uuid: "66f25b3d96fa40f294be341e1266df2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56dec7a4e0b8482aa22edc244ee37c51" is_pre_election: true
I20260620 02:52:53.826310 7480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2328b5e204644b5a95f1828ca1770609" candidate_uuid: "66f25b3d96fa40f294be341e1266df2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33858113a83d4f2da92c8f2364a93c83" is_pre_election: true
I20260620 02:52:53.826563 7480 raft_consensus.cc:2454] T 2328b5e204644b5a95f1828ca1770609 P 33858113a83d4f2da92c8f2364a93c83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66f25b3d96fa40f294be341e1266df2d in term 0.
I20260620 02:52:53.826978 7561 leader_election.cc:302] T 2328b5e204644b5a95f1828ca1770609 P 66f25b3d96fa40f294be341e1266df2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33858113a83d4f2da92c8f2364a93c83, 66f25b3d96fa40f294be341e1266df2d; no voters:
I20260620 02:52:53.827267 7918 raft_consensus.cc:2781] T 2328b5e204644b5a95f1828ca1770609 P 66f25b3d96fa40f294be341e1266df2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:52:53.827400 7918 raft_consensus.cc:482] T 2328b5e204644b5a95f1828ca1770609 P 66f25b3d96fa40f294be341e1266df2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:52:53.827487 7918 raft_consensus.cc:3037] T 2328b5e204644b5a95f1828ca1770609 P 66f25b3d96fa40f294be341e1266df2d [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:53.828999 7910 raft_consensus.cc:504] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56dec7a4e0b8482aa22edc244ee37c51" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43517 } } peers { permanent_uuid: "66f25b3d96fa40f294be341e1266df2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36091 } } peers { permanent_uuid: "33858113a83d4f2da92c8f2364a93c83" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43733 } }
I20260620 02:52:53.829455 7769 raft_consensus.cc:2379] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66f25b3d96fa40f294be341e1266df2d in current term 1: Already voted for candidate 56dec7a4e0b8482aa22edc244ee37c51 in this term.
I20260620 02:52:53.830984 7910 leader_election.cc:288] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51 [CANDIDATE]: Term 1 election: Requested vote from peers 66f25b3d96fa40f294be341e1266df2d (127.0.97.2:36091), 33858113a83d4f2da92c8f2364a93c83 (127.0.97.1:43733)
I20260620 02:52:53.830847 7918 raft_consensus.cc:504] T 2328b5e204644b5a95f1828ca1770609 P 66f25b3d96fa40f294be341e1266df2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56dec7a4e0b8482aa22edc244ee37c51" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43517 } } peers { permanent_uuid: "66f25b3d96fa40f294be341e1266df2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36091 } } peers { permanent_uuid: "33858113a83d4f2da92c8f2364a93c83" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43733 } }
I20260620 02:52:53.830981 7628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2328b5e204644b5a95f1828ca1770609" candidate_uuid: "56dec7a4e0b8482aa22edc244ee37c51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66f25b3d96fa40f294be341e1266df2d"
I20260620 02:52:53.831311 7628 raft_consensus.cc:2379] T 2328b5e204644b5a95f1828ca1770609 P 66f25b3d96fa40f294be341e1266df2d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 56dec7a4e0b8482aa22edc244ee37c51 in current term 1: Already voted for candidate 66f25b3d96fa40f294be341e1266df2d in this term.
I20260620 02:52:53.831284 7480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2328b5e204644b5a95f1828ca1770609" candidate_uuid: "56dec7a4e0b8482aa22edc244ee37c51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33858113a83d4f2da92c8f2364a93c83"
I20260620 02:52:53.831468 7480 raft_consensus.cc:3037] T 2328b5e204644b5a95f1828ca1770609 P 33858113a83d4f2da92c8f2364a93c83 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:53.831471 7918 leader_election.cc:288] T 2328b5e204644b5a95f1828ca1770609 P 66f25b3d96fa40f294be341e1266df2d [CANDIDATE]: Term 1 election: Requested vote from peers 56dec7a4e0b8482aa22edc244ee37c51 (127.0.97.3:43517), 33858113a83d4f2da92c8f2364a93c83 (127.0.97.1:43733)
I20260620 02:52:53.833420 7480 raft_consensus.cc:2454] T 2328b5e204644b5a95f1828ca1770609 P 33858113a83d4f2da92c8f2364a93c83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56dec7a4e0b8482aa22edc244ee37c51 in term 1.
I20260620 02:52:53.833894 7703 leader_election.cc:302] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33858113a83d4f2da92c8f2364a93c83, 56dec7a4e0b8482aa22edc244ee37c51; no voters:
I20260620 02:52:53.834137 7480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2328b5e204644b5a95f1828ca1770609" candidate_uuid: "66f25b3d96fa40f294be341e1266df2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33858113a83d4f2da92c8f2364a93c83"
I20260620 02:52:53.834460 7480 raft_consensus.cc:2379] T 2328b5e204644b5a95f1828ca1770609 P 33858113a83d4f2da92c8f2364a93c83 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 66f25b3d96fa40f294be341e1266df2d in current term 1: Already voted for candidate 56dec7a4e0b8482aa22edc244ee37c51 in this term.
I20260620 02:52:53.834682 7769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2328b5e204644b5a95f1828ca1770609" candidate_uuid: "66f25b3d96fa40f294be341e1266df2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56dec7a4e0b8482aa22edc244ee37c51"
I20260620 02:52:53.834916 7769 raft_consensus.cc:2379] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 66f25b3d96fa40f294be341e1266df2d in current term 1: Already voted for candidate 56dec7a4e0b8482aa22edc244ee37c51 in this term.
I20260620 02:52:53.835217 7910 raft_consensus.cc:2781] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:53.835314 7562 leader_election.cc:302] T 2328b5e204644b5a95f1828ca1770609 P 66f25b3d96fa40f294be341e1266df2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66f25b3d96fa40f294be341e1266df2d; no voters: 33858113a83d4f2da92c8f2364a93c83, 56dec7a4e0b8482aa22edc244ee37c51
I20260620 02:52:53.835661 7918 raft_consensus.cc:2726] T 2328b5e204644b5a95f1828ca1770609 P 66f25b3d96fa40f294be341e1266df2d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 02:52:53.835759 7910 raft_consensus.cc:686] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51 [term 1 LEADER]: Becoming Leader. State: Replica: 56dec7a4e0b8482aa22edc244ee37c51, State: Running, Role: LEADER
I20260620 02:52:53.836623 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.836275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.97.3:60137)
0620 02:52:53.836405 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:53.836496 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 02:52:53.836553 7910 consensus_queue.cc:237] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56dec7a4e0b8482aa22edc244ee37c51" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43517 } } peers { permanent_uuid: "66f25b3d96fa40f294be341e1266df2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36091 } } peers { permanent_uuid: "33858113a83d4f2da92c8f2364a93c83" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43733 } }
I20260620 02:52:53.840370 7156 catalog_manager.cc:5495] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56dec7a4e0b8482aa22edc244ee37c51 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "56dec7a4e0b8482aa22edc244ee37c51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56dec7a4e0b8482aa22edc244ee37c51" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66f25b3d96fa40f294be341e1266df2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33858113a83d4f2da92c8f2364a93c83" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43733 } health_report { overall_health: UNKNOWN } } }
I20260620 02:52:53.885449 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.879331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52150 (local address 127.0.97.3:43517)
0620 02:52:53.879597 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:53.879608 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:53.879632 (+ 24us) server_negotiation.cc:407] Connection header received
0620 02:52:53.879753 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:53.879758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:53.879843 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:53.880040 (+ 197us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:52:53.880460 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:53.881303 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:53.882876 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:53.883393 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:53.883563 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:53.884033 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:52:53.884629 (+ 596us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:52:53.885006 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:53.885089 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 02:52:53.885185 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 02:52:53.890055 7480 raft_consensus.cc:1261] T 2328b5e204644b5a95f1828ca1770609 P 33858113a83d4f2da92c8f2364a93c83 [term 1 FOLLOWER]: Refusing update from remote peer 56dec7a4e0b8482aa22edc244ee37c51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:53.890476 7628 raft_consensus.cc:1261] T 2328b5e204644b5a95f1828ca1770609 P 66f25b3d96fa40f294be341e1266df2d [term 1 FOLLOWER]: Refusing update from remote peer 56dec7a4e0b8482aa22edc244ee37c51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:53.890842 7921 consensus_queue.cc:1035] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33858113a83d4f2da92c8f2364a93c83" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:52:53.891373 7910 consensus_queue.cc:1035] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f25b3d96fa40f294be341e1266df2d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:52:53.894317 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.893622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.97.1:37213)
0620 02:52:53.894006 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:53.894125 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231}
I20260620 02:52:53.894899 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.893724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.97.2:40547)
0620 02:52:53.894610 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:53.894716 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":791}
I20260620 02:52:53.902768 7924 mvcc.cc:205] Tried to move back new op lower bound from 7298760597045846016 to 7298760596838408192. Current Snapshot: MvccSnapshot[applied={T|T < 7298760597045846016}]
I20260620 02:52:53.903558 7926 mvcc.cc:205] Tried to move back new op lower bound from 7298760597045846016 to 7298760596838408192. Current Snapshot: MvccSnapshot[applied={T|T < 7298760597045846016}]
I20260620 02:52:53.904570 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.904020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.97.2:44105)
0620 02:52:53.904298 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:53.904396 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260620 02:52:53.909288 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.908730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.97.1:33035)
0620 02:52:53.908960 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:53.909054 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260620 02:52:53.909624 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.906478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.97.3:45933)
0620 02:52:53.909245 (+ 2767us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:53.909340 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2667}
I20260620 02:52:53.910342 7925 mvcc.cc:205] Tried to move back new op lower bound from 7298760597045846016 to 7298760596838408192. Current Snapshot: MvccSnapshot[applied={T|T < 7298760597045846016}]
I20260620 02:52:53.922679 7156 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:52:53.923014 7156 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:52:53.935386 7156 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 982c8b2c204d443087f6c9f667437974: Sending DeleteTablet for 3 replicas of tablet 2328b5e204644b5a95f1828ca1770609
I20260620 02:52:53.936748 7608 tablet_service.cc:1514] Processing DeleteTablet for tablet 2328b5e204644b5a95f1828ca1770609 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:53 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:44268
I20260620 02:52:53.936754 7460 tablet_service.cc:1514] Processing DeleteTablet for tablet 2328b5e204644b5a95f1828ca1770609 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:53 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:47854
I20260620 02:52:53.937692 7749 tablet_service.cc:1514] Processing DeleteTablet for tablet 2328b5e204644b5a95f1828ca1770609 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:53 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:52148
I20260620 02:52:53.939942 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 7319
I20260620 02:52:53.949292 7935 tablet_replica.cc:330] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51: stopping tablet replica
I20260620 02:52:53.949707 7935 raft_consensus.cc:2229] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:52:53.950525 7935 raft_consensus.cc:2258] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:53.950642 7934 tablet_replica.cc:330] T 2328b5e204644b5a95f1828ca1770609 P 33858113a83d4f2da92c8f2364a93c83: stopping tablet replica
I20260620 02:52:53.951066 7934 raft_consensus.cc:2229] T 2328b5e204644b5a95f1828ca1770609 P 33858113a83d4f2da92c8f2364a93c83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:52:53.951475 7934 raft_consensus.cc:2258] T 2328b5e204644b5a95f1828ca1770609 P 33858113a83d4f2da92c8f2364a93c83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:53.954334 7935 ts_tablet_manager.cc:1905] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:53.959430 7933 tablet_replica.cc:330] T 2328b5e204644b5a95f1828ca1770609 P 66f25b3d96fa40f294be341e1266df2d: stopping tablet replica
I20260620 02:52:53.960599 7933 raft_consensus.cc:2229] T 2328b5e204644b5a95f1828ca1770609 P 66f25b3d96fa40f294be341e1266df2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:52:53.961072 7933 raft_consensus.cc:2258] T 2328b5e204644b5a95f1828ca1770609 P 66f25b3d96fa40f294be341e1266df2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:53.962319 7935 ts_tablet_manager.cc:1918] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:52:53.963027 7934 ts_tablet_manager.cc:1905] T 2328b5e204644b5a95f1828ca1770609 P 33858113a83d4f2da92c8f2364a93c83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:53.962734 7935 log.cc:1196] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-2/wal/wals/2328b5e204644b5a95f1828ca1770609
I20260620 02:52:53.963692 7935 ts_tablet_manager.cc:1939] T 2328b5e204644b5a95f1828ca1770609 P 56dec7a4e0b8482aa22edc244ee37c51: Deleting consensus metadata
I20260620 02:52:53.964437 7933 ts_tablet_manager.cc:1905] T 2328b5e204644b5a95f1828ca1770609 P 66f25b3d96fa40f294be341e1266df2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:53.964953 7142 catalog_manager.cc:4841] TS 56dec7a4e0b8482aa22edc244ee37c51 (127.0.97.3:43517): tablet 2328b5e204644b5a95f1828ca1770609 (table test-table [id=0f39928cb15d4bf69f183fae86b9a734]) successfully deleted
I20260620 02:52:53.968649 7934 ts_tablet_manager.cc:1918] T 2328b5e204644b5a95f1828ca1770609 P 33858113a83d4f2da92c8f2364a93c83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:52:53.968834 7934 log.cc:1196] T 2328b5e204644b5a95f1828ca1770609 P 33858113a83d4f2da92c8f2364a93c83: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-0/wal/wals/2328b5e204644b5a95f1828ca1770609
I20260620 02:52:53.969416 7934 ts_tablet_manager.cc:1939] T 2328b5e204644b5a95f1828ca1770609 P 33858113a83d4f2da92c8f2364a93c83: Deleting consensus metadata
I20260620 02:52:53.970831 7141 catalog_manager.cc:4841] TS 33858113a83d4f2da92c8f2364a93c83 (127.0.97.1:43733): tablet 2328b5e204644b5a95f1828ca1770609 (table test-table [id=0f39928cb15d4bf69f183fae86b9a734]) successfully deleted
I20260620 02:52:53.973541 7933 ts_tablet_manager.cc:1918] T 2328b5e204644b5a95f1828ca1770609 P 66f25b3d96fa40f294be341e1266df2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:52:53.973791 7933 log.cc:1196] T 2328b5e204644b5a95f1828ca1770609 P 66f25b3d96fa40f294be341e1266df2d: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781923914418765-388-0/minicluster-data/ts-1/wal/wals/2328b5e204644b5a95f1828ca1770609
I20260620 02:52:53.974422 7933 ts_tablet_manager.cc:1939] T 2328b5e204644b5a95f1828ca1770609 P 66f25b3d96fa40f294be341e1266df2d: Deleting consensus metadata
I20260620 02:52:53.975845 7141 catalog_manager.cc:4841] TS 66f25b3d96fa40f294be341e1266df2d (127.0.97.2:36091): tablet 2328b5e204644b5a95f1828ca1770609 (table test-table [id=0f39928cb15d4bf69f183fae86b9a734]) successfully deleted
I20260620 02:52:53.980459 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:53.941601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47858 (local address 127.0.97.1:43733)
0620 02:52:53.941819 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:53.941825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:53.953316 (+ 11491us) server_negotiation.cc:407] Connection header received
0620 02:52:53.953527 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:53.953536 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:53.953650 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:53.953792 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:53.956509 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:53.957820 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:53.961281 (+ 3461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:53.961939 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:53.962062 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:53.971494 (+ 9432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:53.971543 (+ 49us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:53.971565 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:53.971629 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:53.975899 (+ 4270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:53.977161 (+ 1262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:53.977239 (+ 78us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:53.977244 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:53.977618 (+ 374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:53.978224 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:53.978230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:53.978234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:53.978864 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:53.979309 (+ 445us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:53.979748 (+ 439us) server_negotiation.cc:299] Negotiation successful
0620 02:52:53.980033 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 02:52:53.981570 7520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:54.083665 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 7319
I20260620 02:52:54.107424 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 7537
I20260620 02:52:54.124128 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:54.108246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44284 (local address 127.0.97.2:36091)
0620 02:52:54.108436 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:54.108443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:54.108464 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:52:54.108650 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:54.108655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:54.108746 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:54.108861 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:54.110457 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:54.111563 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:54.113121 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:54.113745 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:54.113877 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:54.116998 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:54.117018 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:54.117042 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:54.117089 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:54.121088 (+ 3999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:54.121872 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:54.121881 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:54.121885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:54.122036 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:54.122405 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:54.122410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:54.122414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:54.122889 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:54.123207 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:54.123492 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 02:52:54.123750 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 02:52:54.124972 7668 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:54.150622 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:54.149985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.0.1:39320)
0620 02:52:54.150261 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:54.150422 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260620 02:52:54.165689 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:54.165143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45057 (local address 127.0.0.1:39322)
0620 02:52:54.165323 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:54.165499 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 02:52:54.291441 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 7537
I20260620 02:52:54.312927 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 7686
I20260620 02:52:54.327788 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:54.314075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52152 (local address 127.0.97.3:43517)
0620 02:52:54.314344 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:54.314348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:54.314405 (+ 57us) server_negotiation.cc:407] Connection header received
0620 02:52:54.314473 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:54.314476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:54.314545 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:54.314659 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:54.316047 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:54.316917 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:54.318114 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:54.318642 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:54.318783 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:54.321439 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:54.321462 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:54.321477 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:54.321509 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:54.324797 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:54.325445 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:54.325486 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:54.325491 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:54.325684 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:54.326055 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:54.326061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:54.326067 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:54.326633 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:54.326952 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:54.327222 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 02:52:54.327424 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260620 02:52:54.328491 7809 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:54.461642 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 7686
I20260620 02:52:54.481943 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 7018
I20260620 02:52:54.483024 7186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:54.604219 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 7018
I20260620 02:52:54.625030 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 7120
I20260620 02:52:54.626061 7376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:52:54.739842 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 7120
2026-06-20T02:52:54Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9883 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:52:54 dist-test-slave-8p8c krb5kdc[7945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:52:54 dist-test-slave-8p8c krb5kdc[7945](info): set up 2 sockets
Jun 20 02:52:54 dist-test-slave-8p8c krb5kdc[7945](info): commencing operation
krb5kdc: starting...
W20260620 02:52:55.867434 388 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:52:56 dist-test-slave-8p8c krb5kdc[7945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:52:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:52:56Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260620 02:52:56.099264 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39285
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:56.424670 7958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:56.424975 7958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:56.425328 7958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:56.433996 7958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:52:56.434152 7958 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 02:52:56.434224 7958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:56.434320 7958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:56.434414 7958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:52:56.434504 7958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:52:56.445194 7958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39285
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:56.446597 7958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:56.447984 7958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:56.460649 7966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:56.460747 7964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:56.461560 7963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:56.461601 7958 server_base.cc:1034] running on GCE node
I20260620 02:52:56.462626 7958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:56.464119 7958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:56.465374 7958 hybrid_clock.cc:648] HybridClock initialized: now 1781923976465290 us; error 50 us; skew 500 ppm
I20260620 02:52:56.465751 7958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:56 dist-test-slave-8p8c krb5kdc[7945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923976, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:56.469210 7958 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260620 02:52:56.470963 7958 webserver.cc:460] Webserver started at http://127.0.97.62:36589/ using document root <none> and password file <none>
I20260620 02:52:56.471735 7958 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:56.471832 7958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:56.472203 7958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:56.475467 7958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "f4e7c66c5c1845aebeeedfe4dc6532f0"
format_stamp: "Formatted at 2026-06-20 02:52:56 on dist-test-slave-8p8c"
server_key: "0c85b8e0180f3c42c3773c1fce751784"
server_key_iv: "696f5b9f69881f3cddded2d039719b29"
server_key_version: "encryptionkey@0"
I20260620 02:52:56.476509 7958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "f4e7c66c5c1845aebeeedfe4dc6532f0"
format_stamp: "Formatted at 2026-06-20 02:52:56 on dist-test-slave-8p8c"
server_key: "0c85b8e0180f3c42c3773c1fce751784"
server_key_iv: "696f5b9f69881f3cddded2d039719b29"
server_key_version: "encryptionkey@0"
I20260620 02:52:56.482123 7958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:52:56.485805 7973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:56.487516 7958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:52:56.487700 7958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "f4e7c66c5c1845aebeeedfe4dc6532f0"
format_stamp: "Formatted at 2026-06-20 02:52:56 on dist-test-slave-8p8c"
server_key: "0c85b8e0180f3c42c3773c1fce751784"
server_key_iv: "696f5b9f69881f3cddded2d039719b29"
server_key_version: "encryptionkey@0"
I20260620 02:52:56.487917 7958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:56.513039 7958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:56.518102 7958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:56.518452 7958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:56.542577 7958 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39285
I20260620 02:52:56.542631 8034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39285 every 8 connection(s)
I20260620 02:52:56.543946 7958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:52:56.548056 8035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:56.550738 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 7958
I20260620 02:52:56.550958 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:52:56.551622 388 external_mini_cluster.cc:1428] Setting key 26af92ca32251668e95d1635e45f3dae
I20260620 02:52:56.560626 8035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4e7c66c5c1845aebeeedfe4dc6532f0: Bootstrap starting.
I20260620 02:52:56.564355 8035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4e7c66c5c1845aebeeedfe4dc6532f0: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:56.565439 8035 log.cc:824] T 00000000000000000000000000000000 P f4e7c66c5c1845aebeeedfe4dc6532f0: Log is configured to *not* fsync() on all Append() calls
Jun 20 02:52:56 dist-test-slave-8p8c krb5kdc[7945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:52:56.568526 8035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4e7c66c5c1845aebeeedfe4dc6532f0: No bootstrap required, opened a new log
I20260620 02:52:56.575752 8038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:56.553787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45474 (local address 127.0.97.62:39285)
0620 02:52:56.554756 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:56.554771 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:56.554816 (+ 45us) server_negotiation.cc:407] Connection header received
0620 02:52:56.556577 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:56.556604 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:56.557061 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:56.557525 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:56.559089 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:56.560392 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:56.561737 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:56.562576 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:56.562787 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:56.567290 (+ 4503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:56.567298 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:56.567314 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:56.567354 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:56.570661 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:56.571807 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:56.571817 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:56.571824 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:56.571993 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:56.572461 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:56.572465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:56.572467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:56.573238 (+ 771us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:56.573453 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:56.574022 (+ 569us) server_negotiation.cc:299] Negotiation successful
0620 02:52:56.574521 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":758,"thread_start_us":377,"threads_started":1}
I20260620 02:52:56.577916 8035 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4e7c66c5c1845aebeeedfe4dc6532f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e7c66c5c1845aebeeedfe4dc6532f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39285 } }
I20260620 02:52:56.578315 8035 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4e7c66c5c1845aebeeedfe4dc6532f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:56.578490 8035 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4e7c66c5c1845aebeeedfe4dc6532f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4e7c66c5c1845aebeeedfe4dc6532f0, State: Initialized, Role: FOLLOWER
I20260620 02:52:56.579092 8035 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4e7c66c5c1845aebeeedfe4dc6532f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e7c66c5c1845aebeeedfe4dc6532f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39285 } }
I20260620 02:52:56.579300 8035 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4e7c66c5c1845aebeeedfe4dc6532f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:52:56.579424 8035 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4e7c66c5c1845aebeeedfe4dc6532f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:52:56.579564 8035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4e7c66c5c1845aebeeedfe4dc6532f0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:56.582211 8035 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4e7c66c5c1845aebeeedfe4dc6532f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e7c66c5c1845aebeeedfe4dc6532f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39285 } }
I20260620 02:52:56.582755 8035 leader_election.cc:302] T 00000000000000000000000000000000 P f4e7c66c5c1845aebeeedfe4dc6532f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4e7c66c5c1845aebeeedfe4dc6532f0; no voters:
I20260620 02:52:56.583281 8035 leader_election.cc:288] T 00000000000000000000000000000000 P f4e7c66c5c1845aebeeedfe4dc6532f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:52:56.583590 8040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4e7c66c5c1845aebeeedfe4dc6532f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:56.584367 8040 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4e7c66c5c1845aebeeedfe4dc6532f0 [term 1 LEADER]: Becoming Leader. State: Replica: f4e7c66c5c1845aebeeedfe4dc6532f0, State: Running, Role: LEADER
I20260620 02:52:56.584877 8040 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4e7c66c5c1845aebeeedfe4dc6532f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e7c66c5c1845aebeeedfe4dc6532f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39285 } }
I20260620 02:52:56.585647 8035 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4e7c66c5c1845aebeeedfe4dc6532f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:52:56.587829 8042 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4e7c66c5c1845aebeeedfe4dc6532f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4e7c66c5c1845aebeeedfe4dc6532f0. Latest consensus state: current_term: 1 leader_uuid: "f4e7c66c5c1845aebeeedfe4dc6532f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e7c66c5c1845aebeeedfe4dc6532f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39285 } } }
I20260620 02:52:56.588272 8042 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4e7c66c5c1845aebeeedfe4dc6532f0 [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:56.591099 8048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:52:56.591051 8041 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4e7c66c5c1845aebeeedfe4dc6532f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4e7c66c5c1845aebeeedfe4dc6532f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e7c66c5c1845aebeeedfe4dc6532f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39285 } } }
I20260620 02:52:56.591588 8041 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4e7c66c5c1845aebeeedfe4dc6532f0 [sys.catalog]: This master's current role is: LEADER
I20260620 02:52:56.597002 8048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:52:56.607203 8048 catalog_manager.cc:1347] Generated new cluster ID: 008756c39d964c42a0debacde33a78e5
I20260620 02:52:56.607307 8048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:52:56.616375 8048 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:52:56.617779 8048 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:52:56.625381 8048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4e7c66c5c1845aebeeedfe4dc6532f0: Generated new TSK 0
I20260620 02:52:56.626293 8048 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260620 02:52:56.691502 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39285
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:57.026543 8061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:57.026851 8061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:57.026948 8061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:57.034286 8061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:57.034409 8061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:57.034497 8061 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:52:57.034576 8061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:52:57.044365 8061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:57.045621 8061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:57.046958 8061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:57.060570 8066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:57.060506 8067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:57.060461 8069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:57.061489 8061 server_base.cc:1034] running on GCE node
I20260620 02:52:57.062090 8061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:57.062997 8061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:57.064249 8061 hybrid_clock.cc:648] HybridClock initialized: now 1781923977064175 us; error 53 us; skew 500 ppm
I20260620 02:52:57.064553 8061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:57 dist-test-slave-8p8c krb5kdc[7945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923977, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:57.067919 8061 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 02:52:57.069474 8061 webserver.cc:460] Webserver started at http://127.0.97.1:40945/ using document root <none> and password file <none>
I20260620 02:52:57.070147 8061 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:57.070245 8061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:57.070665 8061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:57.073746 8061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-0/data/instance:
uuid: "6b60b7bcf07f449a98435b338b4a952c"
format_stamp: "Formatted at 2026-06-20 02:52:57 on dist-test-slave-8p8c"
server_key: "fe7898e482c60d4be655a9f06ebfc898"
server_key_iv: "981272b1480d73c32121cf0c767af35d"
server_key_version: "encryptionkey@0"
I20260620 02:52:57.074599 8061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6b60b7bcf07f449a98435b338b4a952c"
format_stamp: "Formatted at 2026-06-20 02:52:57 on dist-test-slave-8p8c"
server_key: "fe7898e482c60d4be655a9f06ebfc898"
server_key_iv: "981272b1480d73c32121cf0c767af35d"
server_key_version: "encryptionkey@0"
I20260620 02:52:57.079828 8061 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 02:52:57.083349 8076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:57.084645 8061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 02:52:57.084800 8061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-0/wal
uuid: "6b60b7bcf07f449a98435b338b4a952c"
format_stamp: "Formatted at 2026-06-20 02:52:57 on dist-test-slave-8p8c"
server_key: "fe7898e482c60d4be655a9f06ebfc898"
server_key_iv: "981272b1480d73c32121cf0c767af35d"
server_key_version: "encryptionkey@0"
I20260620 02:52:57.084991 8061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:57.101154 8061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:57.105222 8061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:57.105476 8061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:57.108246 8061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:57.108350 8061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:57.108543 8061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:57.108640 8061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:52:57 dist-test-slave-8p8c krb5kdc[7945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923977, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:52:57.133167 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:57.108632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45131 (local address 127.0.97.62:39285)
0620 02:52:57.109100 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:57.109108 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:57.110216 (+ 1108us) server_negotiation.cc:407] Connection header received
0620 02:52:57.112343 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:57.112351 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:57.112454 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:57.112592 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:57.115133 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:57.116427 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:57.119253 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:57.120046 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:57.120184 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:57.125124 (+ 4940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:57.125134 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:57.125142 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:57.125196 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:57.128691 (+ 3495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:57.129876 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:57.129884 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:57.129888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:57.130046 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:57.130691 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:57.130696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:57.130700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:57.131107 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:57.131258 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:57.132289 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0620 02:52:57.132635 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":240,"threads_started":1}
I20260620 02:52:57.133708 8086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:57.109302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39285 (local address 127.0.97.1:45131)
0620 02:52:57.110064 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:57.110106 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:57.111975 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:57.112994 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:57.113008 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:57.113912 (+ 904us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:57.114903 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:57.114936 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:57.116619 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:57.116626 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:57.119106 (+ 2480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:57.119119 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:57.120383 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:57.120395 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:57.120650 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:57.121610 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:57.121658 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:57.124810 (+ 3152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:57.128878 (+ 4068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:57.128890 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:57.128909 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:57.129709 (+ 800us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:57.130177 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:57.130182 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:57.130190 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:57.130568 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:57.131328 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:57.131334 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:57.131837 (+ 503us) client_negotiation.cc:769] Sending connection context
0620 02:52:57.132078 (+ 241us) client_negotiation.cc:240] Negotiation successful
0620 02:52:57.132479 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":305,"threads_started":1}
I20260620 02:52:57.155915 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:57.141630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39285 (local address 127.0.97.1:59179)
0620 02:52:57.142292 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:57.142308 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:57.142522 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:57.142877 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:57.142881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:57.143335 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:57.143618 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:57.143627 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:57.144605 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:57.144610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:57.145603 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:57.145613 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:57.146329 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:57.146335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:57.146550 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:57.147285 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:57.147312 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:57.148713 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:57.152663 (+ 3950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:57.152673 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:57.152677 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:57.153433 (+ 756us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:57.153930 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:57.153938 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:57.153944 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:57.154143 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:57.154817 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:57.154821 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:57.154994 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 02:52:57.155119 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 02:52:57.155565 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":667,"threads_started":1}
I20260620 02:52:57.156044 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:57.141993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59179 (local address 127.0.97.62:39285)
0620 02:52:57.142295 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:57.142304 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:57.142431 (+ 127us) server_negotiation.cc:407] Connection header received
0620 02:52:57.142657 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:57.142663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:57.142749 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:57.142853 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:57.143760 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:57.144471 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:57.145745 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:57.146147 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:57.146284 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:57.148909 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:57.148922 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:57.148926 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:57.148955 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:57.152475 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:57.153612 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:57.153622 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:57.153628 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:57.153791 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:57.154272 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:57.154277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:57.154280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:57.154697 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:57.154859 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:57.155287 (+ 428us) server_negotiation.cc:299] Negotiation successful
0620 02:52:57.155550 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260620 02:52:57.166060 8061 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36377
I20260620 02:52:57.166096 8196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36377 every 8 connection(s)
I20260620 02:52:57.167336 8061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:52:57.169524 8197 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39285
I20260620 02:52:57.169767 8197 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:57.170450 8197 heartbeater.cc:511] Master 127.0.97.62:39285 requested a full tablet report, sending...
I20260620 02:52:57.172505 7990 ts_manager.cc:194] Registered new tserver with Master: 6b60b7bcf07f449a98435b338b4a952c (127.0.97.1:36377)
I20260620 02:52:57.173128 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 8061
I20260620 02:52:57.173312 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:52:57.174083 388 external_mini_cluster.cc:1428] Setting key d452b2cea8ec2761cc7f83da4495e2b2
I20260620 02:52:57.174677 7990 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45131
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260620 02:52:57.222204 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39285
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:57.555213 8201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:57.555526 8201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:57.555703 8201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:57.563126 8201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:57.563269 8201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:57.563426 8201 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:52:57.563565 8201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:52:57.573333 8201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:57.574688 8201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:57.576014 8201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:57.589474 8207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:57.589501 8206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:57.589676 8209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:57.590451 8201 server_base.cc:1034] running on GCE node
I20260620 02:52:57.591068 8201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:57.592171 8201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:57.593422 8201 hybrid_clock.cc:648] HybridClock initialized: now 1781923977593329 us; error 72 us; skew 500 ppm
I20260620 02:52:57.593739 8201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:57 dist-test-slave-8p8c krb5kdc[7945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923977, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:57.597093 8201 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 02:52:57.598747 8201 webserver.cc:460] Webserver started at http://127.0.97.2:34177/ using document root <none> and password file <none>
I20260620 02:52:57.599454 8201 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:57.599578 8201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:57.599910 8201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:57.602851 8201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-1/data/instance:
uuid: "8eeb98eed5c94cc898e55490f1ebd0cb"
format_stamp: "Formatted at 2026-06-20 02:52:57 on dist-test-slave-8p8c"
server_key: "15d4794a6b16f6dc232b8d28bba58a49"
server_key_iv: "ab1c5ab19864fc4155641ff7edd14d74"
server_key_version: "encryptionkey@0"
I20260620 02:52:57.603727 8201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8eeb98eed5c94cc898e55490f1ebd0cb"
format_stamp: "Formatted at 2026-06-20 02:52:57 on dist-test-slave-8p8c"
server_key: "15d4794a6b16f6dc232b8d28bba58a49"
server_key_iv: "ab1c5ab19864fc4155641ff7edd14d74"
server_key_version: "encryptionkey@0"
I20260620 02:52:57.608891 8201 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:52:57.612434 8216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:57.613823 8201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:52:57.614001 8201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-1/wal
uuid: "8eeb98eed5c94cc898e55490f1ebd0cb"
format_stamp: "Formatted at 2026-06-20 02:52:57 on dist-test-slave-8p8c"
server_key: "15d4794a6b16f6dc232b8d28bba58a49"
server_key_iv: "ab1c5ab19864fc4155641ff7edd14d74"
server_key_version: "encryptionkey@0"
I20260620 02:52:57.614233 8201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:57.652907 8201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:57.658097 8201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:57.658525 8201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:57.661391 8201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:57.661517 8201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:57.661691 8201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:57.661798 8201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:52:57 dist-test-slave-8p8c krb5kdc[7945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923977, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:52:57.687383 8226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:57.663852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32935 (local address 127.0.97.62:39285)
0620 02:52:57.664324 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:57.664331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:57.664523 (+ 192us) server_negotiation.cc:407] Connection header received
0620 02:52:57.666681 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:57.666686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:57.666803 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:57.666924 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:57.669913 (+ 2989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:57.671070 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:57.673788 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:57.674721 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:57.674909 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:57.680751 (+ 5842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:57.680756 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:57.680760 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:57.680797 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:57.684005 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:57.684768 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:57.684775 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:57.684778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:57.684914 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:57.685353 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:57.685359 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:57.685363 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:57.685651 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:57.685799 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:57.686799 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0620 02:52:57.687009 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":1120,"threads_started":1}
I20260620 02:52:57.688521 8225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:57.663342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39285 (local address 127.0.97.2:32935)
0620 02:52:57.664355 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:57.664403 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:57.666309 (+ 1906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:57.667377 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:57.667392 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:57.668421 (+ 1029us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:57.669656 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:57.669697 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:57.671267 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:57.671271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:57.673625 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:57.673638 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:57.675046 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:57.675057 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:57.675364 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:57.676561 (+ 1197us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:57.676587 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:57.680410 (+ 3823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:57.684157 (+ 3747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:57.684168 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:57.684187 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:57.684625 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:57.685034 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:57.685050 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:57.685057 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:57.685235 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:57.685787 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:57.685796 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:57.686333 (+ 537us) client_negotiation.cc:769] Sending connection context
0620 02:52:57.686711 (+ 378us) client_negotiation.cc:240] Negotiation successful
0620 02:52:57.687283 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":528,"threads_started":1}
I20260620 02:52:57.714099 8226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:57.695521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49397 (local address 127.0.97.62:39285)
0620 02:52:57.695743 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:57.695748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:57.696300 (+ 552us) server_negotiation.cc:407] Connection header received
0620 02:52:57.696524 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:57.696530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:57.696611 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:57.696714 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:57.698505 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:57.699531 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:57.702587 (+ 3056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:57.703338 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:57.703507 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:57.706719 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:57.706728 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:57.706734 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:57.706779 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:57.710706 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:57.711837 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:57.711845 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:57.711852 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:57.712019 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:57.712449 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:57.712456 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:57.712459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:57.712800 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:57.712961 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:57.713500 (+ 539us) server_negotiation.cc:299] Negotiation successful
0620 02:52:57.713744 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260620 02:52:57.714208 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:57.695551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39285 (local address 127.0.97.2:49397)
0620 02:52:57.696187 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:57.696205 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:57.696378 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:57.696919 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:57.696924 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:57.697746 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:57.698215 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:57.698233 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:57.699702 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:57.699710 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:57.702295 (+ 2585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:57.702315 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:57.703873 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:57.703879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:57.704155 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:57.704954 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:57.704987 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:57.706409 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:57.710892 (+ 4483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:57.710899 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:57.710903 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:57.711680 (+ 777us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:57.712171 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:57.712178 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:57.712185 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:57.712320 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:57.712925 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:57.712929 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:57.713167 (+ 238us) client_negotiation.cc:769] Sending connection context
0620 02:52:57.713405 (+ 238us) client_negotiation.cc:240] Negotiation successful
0620 02:52:57.713772 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":366,"threads_started":1}
I20260620 02:52:57.727399 8201 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40485
I20260620 02:52:57.727504 8336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40485 every 8 connection(s)
I20260620 02:52:57.728718 8201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:52:57.731295 8337 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39285
I20260620 02:52:57.731536 8337 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:57.732249 8337 heartbeater.cc:511] Master 127.0.97.62:39285 requested a full tablet report, sending...
I20260620 02:52:57.733831 7989 ts_manager.cc:194] Registered new tserver with Master: 8eeb98eed5c94cc898e55490f1ebd0cb (127.0.97.2:40485)
I20260620 02:52:57.734983 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 8201
I20260620 02:52:57.735035 7989 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:32935
I20260620 02:52:57.735172 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:52:57.735793 388 external_mini_cluster.cc:1428] Setting key 3ffe5360413cdcf60901a702918fa063
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260620 02:52:57.785101 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39285
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:52:58.115370 8341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:52:58.115654 8341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:52:58.115814 8341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:52:58.123111 8341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:52:58.123219 8341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:52:58.123396 8341 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 02:52:58.123509 8341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:52:58.133270 8341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:52:58.134713 8341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:52:58.136067 8341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:52:58.149749 8349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:58.149829 8346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:52:58.149857 8347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:52:58.151085 8341 server_base.cc:1034] running on GCE node
I20260620 02:52:58.151716 8341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:52:58.152686 8341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:52:58.153960 8341 hybrid_clock.cc:648] HybridClock initialized: now 1781923978153895 us; error 40 us; skew 500 ppm
I20260620 02:52:58.154281 8341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:52:58 dist-test-slave-8p8c krb5kdc[7945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923978, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:52:58.157804 8341 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 02:52:58.159475 8341 webserver.cc:460] Webserver started at http://127.0.97.3:44517/ using document root <none> and password file <none>
I20260620 02:52:58.160248 8341 fs_manager.cc:359] Metadata directory not provided
I20260620 02:52:58.160385 8341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:52:58.160750 8341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:52:58.164005 8341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-2/data/instance:
uuid: "03703e96a41b434ab9cd6c41f16914bb"
format_stamp: "Formatted at 2026-06-20 02:52:58 on dist-test-slave-8p8c"
server_key: "03dec7e7a1ee6a3c3aa59d5cef0419b2"
server_key_iv: "749811b2b8c33dd6531b1f983dd88579"
server_key_version: "encryptionkey@0"
I20260620 02:52:58.164883 8341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-2/wal/instance:
uuid: "03703e96a41b434ab9cd6c41f16914bb"
format_stamp: "Formatted at 2026-06-20 02:52:58 on dist-test-slave-8p8c"
server_key: "03dec7e7a1ee6a3c3aa59d5cef0419b2"
server_key_iv: "749811b2b8c33dd6531b1f983dd88579"
server_key_version: "encryptionkey@0"
I20260620 02:52:58.170925 8341 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260620 02:52:58.175266 8356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:58.176759 8341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260620 02:52:58.176947 8341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-2/wal
uuid: "03703e96a41b434ab9cd6c41f16914bb"
format_stamp: "Formatted at 2026-06-20 02:52:58 on dist-test-slave-8p8c"
server_key: "03dec7e7a1ee6a3c3aa59d5cef0419b2"
server_key_iv: "749811b2b8c33dd6531b1f983dd88579"
server_key_version: "encryptionkey@0"
I20260620 02:52:58.177183 8341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:52:58.178162 8197 heartbeater.cc:503] Master 127.0.97.62:39285 was elected leader, sending a full tablet report...
I20260620 02:52:58.198832 8341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:52:58.204073 8341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:52:58.204398 8341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:52:58.207553 8341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:52:58.207674 8341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:52:58.207840 8341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:52:58.207937 8341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:52:58 dist-test-slave-8p8c krb5kdc[7945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923978, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:52:58.232626 8226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:58.208140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49501 (local address 127.0.97.62:39285)
0620 02:52:58.208330 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:58.208336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:58.209774 (+ 1438us) server_negotiation.cc:407] Connection header received
0620 02:52:58.211912 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:58.211919 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:58.212026 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:58.212178 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:58.214072 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.215604 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.218813 (+ 3209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.219486 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.219631 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:58.225250 (+ 5619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:58.225258 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:58.225265 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:58.225312 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:58.228836 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:58.229963 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:58.229971 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:58.229976 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:58.230136 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:58.230766 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:58.230774 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:58.230779 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:58.231111 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:58.231279 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:58.232026 (+ 747us) server_negotiation.cc:299] Negotiation successful
0620 02:52:58.232253 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 02:52:58.233734 8365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:58.208716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39285 (local address 127.0.97.3:49501)
0620 02:52:58.209573 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:58.209624 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:58.211612 (+ 1988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:58.212546 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:58.212562 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:58.213267 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:58.213842 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:58.213859 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.215845 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.215853 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:58.218639 (+ 2786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:58.218656 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.219758 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.219766 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:58.220144 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:58.221020 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:58.221064 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:58.224898 (+ 3834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:58.228995 (+ 4097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:58.229005 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:58.229023 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:58.229748 (+ 725us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:58.230265 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:58.230274 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:58.230279 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:58.230572 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:58.231241 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:58.231250 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:58.231637 (+ 387us) client_negotiation.cc:769] Sending connection context
0620 02:52:58.231918 (+ 281us) client_negotiation.cc:240] Negotiation successful
0620 02:52:58.232483 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":382,"threads_started":1}
I20260620 02:52:58.255349 8226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:58.240192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58971 (local address 127.0.97.62:39285)
0620 02:52:58.240446 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:58.240453 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:58.240981 (+ 528us) server_negotiation.cc:407] Connection header received
0620 02:52:58.241124 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:58.241128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:58.241186 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:58.241269 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:58.242844 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.243904 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.245861 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.246271 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.246428 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:58.249041 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:58.249045 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:58.249048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:58.249080 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:58.252145 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:58.253202 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:58.253205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:58.253208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:58.253326 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:58.253700 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:58.253708 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:58.253711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:58.253977 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:58.254153 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:58.254758 (+ 605us) server_negotiation.cc:299] Negotiation successful
0620 02:52:58.254974 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 02:52:58.255472 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:58.240083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39285 (local address 127.0.97.3:58971)
0620 02:52:58.240803 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:58.240828 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:58.241006 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:58.241333 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:58.241341 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:58.242105 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:58.242670 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:58.242685 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.244091 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.244104 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:58.245720 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:58.245732 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.246500 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.246508 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:58.246675 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:58.247495 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:58.247515 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:58.248816 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:58.252343 (+ 3527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:58.252350 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:58.252357 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:58.253028 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:58.253441 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:58.253446 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:58.253449 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:58.253585 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:58.254229 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:58.254233 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:58.254494 (+ 261us) client_negotiation.cc:769] Sending connection context
0620 02:52:58.254673 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 02:52:58.255003 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":356,"threads_started":1}
I20260620 02:52:58.261628 8341 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35929
I20260620 02:52:58.261662 8475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35929 every 8 connection(s)
I20260620 02:52:58.262928 8341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:52:58.265076 8476 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39285
I20260620 02:52:58.265308 8476 heartbeater.cc:464] Registering TS with master...
I20260620 02:52:58.265941 8476 heartbeater.cc:511] Master 127.0.97.62:39285 requested a full tablet report, sending...
I20260620 02:52:58.267371 7989 ts_manager.cc:194] Registered new tserver with Master: 03703e96a41b434ab9cd6c41f16914bb (127.0.97.3:35929)
I20260620 02:52:58.268191 7989 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49501
I20260620 02:52:58.269191 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 8341
I20260620 02:52:58.269374 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:52:58.269811 388 external_mini_cluster.cc:1428] Setting key 29f4edcd8bc44016108fb776c52e3398
I20260620 02:52:58.272553 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 02:52:58 dist-test-slave-8p8c krb5kdc[7945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923978, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 02:52:58 dist-test-slave-8p8c krb5kdc[7945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:52:58 dist-test-slave-8p8c krb5kdc[7945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 02:52:58.322496 8226 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:52:58.311884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45488 (local address 127.0.97.62:39285)
0620 02:52:58.312054 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:58.312058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:58.312416 (+ 358us) server_negotiation.cc:407] Connection header received
0620 02:52:58.312569 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:58.312573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:58.312636 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:58.312729 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:58.314035 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.314913 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.316126 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.316523 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.316617 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:58.322142 (+ 5525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45488: BlockingRecv error: recv got EOF from 127.0.0.1:45488 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 02:52:58 dist-test-slave-8p8c krb5kdc[7945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923978, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:52:58.339390 8226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:58.325753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45500 (local address 127.0.97.62:39285)
0620 02:52:58.325906 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:58.325911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:58.326274 (+ 363us) server_negotiation.cc:407] Connection header received
0620 02:52:58.326438 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:58.326442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:58.326516 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:58.326600 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:58.327692 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.328400 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.329691 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.330122 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.330217 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:58.333972 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:58.333978 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:58.333981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:58.334014 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:58.337016 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:58.337697 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:58.337701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:58.337703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:58.337821 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:58.338150 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:58.338153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:58.338155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:58.338457 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:58.338653 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:58.338884 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 02:52:58.339072 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 02:52:58.343374 7989 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:52:58.350208 7989 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:52:58.374269 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:58.363163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42746 (local address 127.0.97.2:40485)
0620 02:52:58.364324 (+ 1161us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:58.364336 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:58.366198 (+ 1862us) server_negotiation.cc:407] Connection header received
0620 02:52:58.366527 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:58.366536 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:58.366779 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:58.366936 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:58.367822 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.369130 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.371424 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.372797 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.372926 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:58.373151 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:58.373786 (+ 635us) server_negotiation.cc:299] Negotiation successful
0620 02:52:58.373989 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":975,"thread_start_us":872,"threads_started":1}
I20260620 02:52:58.374485 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:58.362911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38360 (local address 127.0.97.3:35929)
0620 02:52:58.364937 (+ 2026us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:58.364945 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:58.365752 (+ 807us) server_negotiation.cc:407] Connection header received
0620 02:52:58.365858 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:58.365867 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:58.366143 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:58.366398 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:58.367138 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.368554 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.371667 (+ 3113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.373097 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.373224 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:58.373444 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:58.373943 (+ 499us) server_negotiation.cc:299] Negotiation successful
0620 02:52:58.374124 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1831,"thread_start_us":1576,"threads_started":1}
I20260620 02:52:58.374998 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:58.362682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35929 (local address 127.0.0.1:38360)
0620 02:52:58.365360 (+ 2678us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:58.365409 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:58.365573 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:58.366428 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:58.366437 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:58.366470 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:58.366879 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:58.366898 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.369042 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.369049 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:58.371492 (+ 2443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:58.371506 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.373595 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.373601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:58.373785 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:58.373815 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 02:52:58.373896 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 02:52:58.374085 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2416,"mutex_wait_us":85,"thread_start_us":2429,"threads_started":1}
I20260620 02:52:58.375173 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:58.362705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40485 (local address 127.0.0.1:42746)
0620 02:52:58.366091 (+ 3386us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:58.366112 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:58.366314 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:58.367166 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:58.367172 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:58.367198 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:58.367610 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:58.367621 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.369296 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.369301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:58.371269 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:58.371280 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.373294 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.373300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:58.373500 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:58.373572 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 02:52:58.374407 (+ 835us) client_negotiation.cc:240] Negotiation successful
0620 02:52:58.374566 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3255,"mutex_wait_us":25,"thread_start_us":2214,"threads_started":1}
I20260620 02:52:58.375486 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:58.365722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36377 (local address 127.0.0.1:36408)
0620 02:52:58.367179 (+ 1457us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:58.367201 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:58.367347 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:58.368362 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:58.368366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:58.368395 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:58.368718 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:58.368729 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.370489 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.370494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:58.372282 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:58.372291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.374210 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.374217 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:58.374839 (+ 622us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:58.374866 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 02:52:58.374973 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 02:52:58.375096 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1307,"mutex_wait_us":709,"thread_start_us":470,"threads_started":1}
I20260620 02:52:58.375558 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:58.365984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36408 (local address 127.0.97.1:36377)
0620 02:52:58.367829 (+ 1845us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:58.367837 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:58.367862 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:52:58.367952 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:58.367969 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:58.368173 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:58.368353 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:58.368864 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.370342 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.372426 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.373752 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.373882 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:58.374063 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:58.375086 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0620 02:52:58.375261 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1704,"thread_start_us":1585,"threads_started":1}
I20260620 02:52:58.379050 8267 tablet_service.cc:1467] Processing CreateTablet for tablet 941e1ab5437540d1b27be85551498332 (DEFAULT_TABLE table=test-table [id=418b2d24fccb44b1834ba9d885e7a980]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:58.379668 8127 tablet_service.cc:1467] Processing CreateTablet for tablet 941e1ab5437540d1b27be85551498332 (DEFAULT_TABLE table=test-table [id=418b2d24fccb44b1834ba9d885e7a980]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:58.380256 8267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 941e1ab5437540d1b27be85551498332. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:58.380831 8127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 941e1ab5437540d1b27be85551498332. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:58.386580 8406 tablet_service.cc:1467] Processing CreateTablet for tablet 941e1ab5437540d1b27be85551498332 (DEFAULT_TABLE table=test-table [id=418b2d24fccb44b1834ba9d885e7a980]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:58.388005 8406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 941e1ab5437540d1b27be85551498332. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:58.393764 8501 tablet_bootstrap.cc:492] T 941e1ab5437540d1b27be85551498332 P 8eeb98eed5c94cc898e55490f1ebd0cb: Bootstrap starting.
I20260620 02:52:58.395359 8500 tablet_bootstrap.cc:492] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c: Bootstrap starting.
I20260620 02:52:58.397863 8501 tablet_bootstrap.cc:654] T 941e1ab5437540d1b27be85551498332 P 8eeb98eed5c94cc898e55490f1ebd0cb: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:58.399174 8501 log.cc:824] T 941e1ab5437540d1b27be85551498332 P 8eeb98eed5c94cc898e55490f1ebd0cb: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:58.400285 8500 tablet_bootstrap.cc:654] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:58.401489 8500 log.cc:824] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:58.402559 8501 tablet_bootstrap.cc:492] T 941e1ab5437540d1b27be85551498332 P 8eeb98eed5c94cc898e55490f1ebd0cb: No bootstrap required, opened a new log
I20260620 02:52:58.402853 8501 ts_tablet_manager.cc:1397] T 941e1ab5437540d1b27be85551498332 P 8eeb98eed5c94cc898e55490f1ebd0cb: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260620 02:52:58.403996 8504 tablet_bootstrap.cc:492] T 941e1ab5437540d1b27be85551498332 P 03703e96a41b434ab9cd6c41f16914bb: Bootstrap starting.
I20260620 02:52:58.404546 8500 tablet_bootstrap.cc:492] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c: No bootstrap required, opened a new log
I20260620 02:52:58.404799 8500 ts_tablet_manager.cc:1397] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260620 02:52:58.407480 8504 tablet_bootstrap.cc:654] T 941e1ab5437540d1b27be85551498332 P 03703e96a41b434ab9cd6c41f16914bb: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:58.408691 8504 log.cc:824] T 941e1ab5437540d1b27be85551498332 P 03703e96a41b434ab9cd6c41f16914bb: Log is configured to *not* fsync() on all Append() calls
I20260620 02:52:58.411087 8501 raft_consensus.cc:348] T 941e1ab5437540d1b27be85551498332 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } }
I20260620 02:52:58.411509 8501 raft_consensus.cc:374] T 941e1ab5437540d1b27be85551498332 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:58.411598 8501 raft_consensus.cc:729] T 941e1ab5437540d1b27be85551498332 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eeb98eed5c94cc898e55490f1ebd0cb, State: Initialized, Role: FOLLOWER
I20260620 02:52:58.411793 8504 tablet_bootstrap.cc:492] T 941e1ab5437540d1b27be85551498332 P 03703e96a41b434ab9cd6c41f16914bb: No bootstrap required, opened a new log
I20260620 02:52:58.412046 8504 ts_tablet_manager.cc:1397] T 941e1ab5437540d1b27be85551498332 P 03703e96a41b434ab9cd6c41f16914bb: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 02:52:58.412324 8501 consensus_queue.cc:260] T 941e1ab5437540d1b27be85551498332 P 8eeb98eed5c94cc898e55490f1ebd0cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } }
I20260620 02:52:58.413451 8337 heartbeater.cc:503] Master 127.0.97.62:39285 was elected leader, sending a full tablet report...
I20260620 02:52:58.413964 8501 ts_tablet_manager.cc:1428] T 941e1ab5437540d1b27be85551498332 P 8eeb98eed5c94cc898e55490f1ebd0cb: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260620 02:52:58.414966 8500 raft_consensus.cc:348] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } }
I20260620 02:52:58.415608 8500 raft_consensus.cc:374] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:58.415735 8500 raft_consensus.cc:729] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b60b7bcf07f449a98435b338b4a952c, State: Initialized, Role: FOLLOWER
I20260620 02:52:58.416375 8500 consensus_queue.cc:260] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } }
I20260620 02:52:58.418057 8500 ts_tablet_manager.cc:1428] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260620 02:52:58.419718 8198 tablet.cc:2369] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:52:58.422776 8504 raft_consensus.cc:348] T 941e1ab5437540d1b27be85551498332 P 03703e96a41b434ab9cd6c41f16914bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } }
I20260620 02:52:58.423137 8504 raft_consensus.cc:374] T 941e1ab5437540d1b27be85551498332 P 03703e96a41b434ab9cd6c41f16914bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:58.423235 8504 raft_consensus.cc:729] T 941e1ab5437540d1b27be85551498332 P 03703e96a41b434ab9cd6c41f16914bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03703e96a41b434ab9cd6c41f16914bb, State: Initialized, Role: FOLLOWER
I20260620 02:52:58.423811 8504 consensus_queue.cc:260] T 941e1ab5437540d1b27be85551498332 P 03703e96a41b434ab9cd6c41f16914bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } }
I20260620 02:52:58.424867 8476 heartbeater.cc:503] Master 127.0.97.62:39285 was elected leader, sending a full tablet report...
I20260620 02:52:58.425520 8504 ts_tablet_manager.cc:1428] T 941e1ab5437540d1b27be85551498332 P 03703e96a41b434ab9cd6c41f16914bb: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
W20260620 02:52:58.480778 8338 tablet.cc:2369] T 941e1ab5437540d1b27be85551498332 P 8eeb98eed5c94cc898e55490f1ebd0cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:52:58.514605 8477 tablet.cc:2369] T 941e1ab5437540d1b27be85551498332 P 03703e96a41b434ab9cd6c41f16914bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:52:58.536567 8507 raft_consensus.cc:482] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:52:58.537006 8507 raft_consensus.cc:504] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } }
I20260620 02:52:58.538736 8507 leader_election.cc:288] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03703e96a41b434ab9cd6c41f16914bb (127.0.97.3:35929), 8eeb98eed5c94cc898e55490f1ebd0cb (127.0.97.2:40485)
I20260620 02:52:58.544221 8506 raft_consensus.cc:482] T 941e1ab5437540d1b27be85551498332 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:52:58.544538 8506 raft_consensus.cc:504] T 941e1ab5437540d1b27be85551498332 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } }
I20260620 02:52:58.545740 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:58.539235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49305 (local address 127.0.97.3:35929)
0620 02:52:58.539440 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:58.539450 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:58.539916 (+ 466us) server_negotiation.cc:407] Connection header received
0620 02:52:58.540188 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:58.540193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:58.540252 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:58.540351 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:58.540851 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.541542 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.543411 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.544633 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.544766 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:58.544873 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:58.545370 (+ 497us) server_negotiation.cc:299] Negotiation successful
0620 02:52:58.545496 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:52:58.546460 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:58.539054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35929 (local address 127.0.97.1:49305)
0620 02:52:58.539802 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:58.539818 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:58.540023 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:58.540370 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:58.540374 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:58.540405 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:58.540719 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:58.540727 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.541669 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.541676 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:58.543265 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:58.543276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.545021 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.545027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:58.545186 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:58.545219 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 02:52:58.545971 (+ 752us) client_negotiation.cc:240] Negotiation successful
0620 02:52:58.546100 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":949,"threads_started":1}
I20260620 02:52:58.546608 8506 leader_election.cc:288] T 941e1ab5437540d1b27be85551498332 P 8eeb98eed5c94cc898e55490f1ebd0cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03703e96a41b434ab9cd6c41f16914bb (127.0.97.3:35929), 6b60b7bcf07f449a98435b338b4a952c (127.0.97.1:36377)
I20260620 02:52:58.546962 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:58.539309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50009 (local address 127.0.97.2:40485)
0620 02:52:58.539560 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:58.539566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:58.540364 (+ 798us) server_negotiation.cc:407] Connection header received
0620 02:52:58.540619 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:58.540628 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:58.540683 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:58.540787 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:58.541274 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.542125 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.544548 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.545785 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.545890 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:58.545992 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:58.546585 (+ 593us) server_negotiation.cc:299] Negotiation successful
0620 02:52:58.546734 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 02:52:58.547622 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:58.539110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40485 (local address 127.0.97.1:50009)
0620 02:52:58.540258 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:58.540275 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:58.540478 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:58.540799 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:58.540803 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:58.540824 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:58.541127 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:58.541135 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.542282 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.542290 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:58.544369 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:58.544383 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.546185 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.546190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:58.546397 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:58.546438 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 02:52:58.547113 (+ 675us) client_negotiation.cc:240] Negotiation successful
0620 02:52:58.547247 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1065,"spinlock_wait_cycles":39296,"thread_start_us":391,"threads_started":1}
I20260620 02:52:58.547506 8426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "941e1ab5437540d1b27be85551498332" candidate_uuid: "6b60b7bcf07f449a98435b338b4a952c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03703e96a41b434ab9cd6c41f16914bb" is_pre_election: true
I20260620 02:52:58.548089 8426 raft_consensus.cc:2454] T 941e1ab5437540d1b27be85551498332 P 03703e96a41b434ab9cd6c41f16914bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b60b7bcf07f449a98435b338b4a952c in term 0.
I20260620 02:52:58.548777 8080 leader_election.cc:302] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03703e96a41b434ab9cd6c41f16914bb, 6b60b7bcf07f449a98435b338b4a952c; no voters:
I20260620 02:52:58.548594 8292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "941e1ab5437540d1b27be85551498332" candidate_uuid: "6b60b7bcf07f449a98435b338b4a952c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" is_pre_election: true
I20260620 02:52:58.549163 8507 raft_consensus.cc:2781] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:52:58.549124 8292 raft_consensus.cc:2454] T 941e1ab5437540d1b27be85551498332 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b60b7bcf07f449a98435b338b4a952c in term 0.
I20260620 02:52:58.549278 8507 raft_consensus.cc:482] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:52:58.549361 8507 raft_consensus.cc:3037] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:58.552465 8507 raft_consensus.cc:504] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } }
I20260620 02:52:58.553445 8426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "941e1ab5437540d1b27be85551498332" candidate_uuid: "6b60b7bcf07f449a98435b338b4a952c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03703e96a41b434ab9cd6c41f16914bb"
I20260620 02:52:58.553637 8426 raft_consensus.cc:3037] T 941e1ab5437540d1b27be85551498332 P 03703e96a41b434ab9cd6c41f16914bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:58.556726 8426 raft_consensus.cc:2454] T 941e1ab5437540d1b27be85551498332 P 03703e96a41b434ab9cd6c41f16914bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b60b7bcf07f449a98435b338b4a952c in term 1.
I20260620 02:52:58.557129 8080 leader_election.cc:302] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03703e96a41b434ab9cd6c41f16914bb, 6b60b7bcf07f449a98435b338b4a952c; no voters:
I20260620 02:52:58.560160 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:58.547583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53783 (local address 127.0.97.3:35929)
0620 02:52:58.547758 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:58.547769 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:58.550969 (+ 3200us) server_negotiation.cc:407] Connection header received
0620 02:52:58.551238 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:58.551242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:58.551320 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:58.551380 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:58.552240 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.553252 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.556802 (+ 3550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.558529 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.558647 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:58.558786 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:58.559607 (+ 821us) server_negotiation.cc:299] Negotiation successful
0620 02:52:58.559793 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 02:52:58.561306 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:58.547532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35929 (local address 127.0.97.2:53783)
0620 02:52:58.550878 (+ 3346us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:58.550895 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:58.551085 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:58.551581 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:58.551588 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:58.551622 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:58.552081 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:58.552095 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.553491 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.553499 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:58.556580 (+ 3081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:58.556600 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.559125 (+ 2525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.559134 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:58.559382 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:58.559434 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 02:52:58.560600 (+ 1166us) client_negotiation.cc:240] Negotiation successful
0620 02:52:58.560804 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3196,"thread_start_us":3046,"threads_started":1}
I20260620 02:52:58.562500 8426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "941e1ab5437540d1b27be85551498332" candidate_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03703e96a41b434ab9cd6c41f16914bb" is_pre_election: true
I20260620 02:52:58.562808 8426 raft_consensus.cc:2379] T 941e1ab5437540d1b27be85551498332 P 03703e96a41b434ab9cd6c41f16914bb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8eeb98eed5c94cc898e55490f1ebd0cb in current term 1: Already voted for candidate 6b60b7bcf07f449a98435b338b4a952c in this term.
I20260620 02:52:58.563524 8292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "941e1ab5437540d1b27be85551498332" candidate_uuid: "6b60b7bcf07f449a98435b338b4a952c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb"
I20260620 02:52:58.563730 8292 raft_consensus.cc:3037] T 941e1ab5437540d1b27be85551498332 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:58.564100 8507 leader_election.cc:288] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c [CANDIDATE]: Term 1 election: Requested vote from peers 03703e96a41b434ab9cd6c41f16914bb (127.0.97.3:35929), 8eeb98eed5c94cc898e55490f1ebd0cb (127.0.97.2:40485)
I20260620 02:52:58.564513 8507 raft_consensus.cc:2781] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:58.564934 8507 raft_consensus.cc:686] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c [term 1 LEADER]: Becoming Leader. State: Replica: 6b60b7bcf07f449a98435b338b4a952c, State: Running, Role: LEADER
I20260620 02:52:58.565665 8507 consensus_queue.cc:237] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } }
I20260620 02:52:58.567047 8292 raft_consensus.cc:2454] T 941e1ab5437540d1b27be85551498332 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b60b7bcf07f449a98435b338b4a952c in term 1.
I20260620 02:52:58.568758 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:58.550950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36377 (local address 127.0.97.2:59141)
0620 02:52:58.561781 (+ 10831us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:58.561803 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:58.561977 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:58.562488 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:58.562493 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:58.562520 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:58.562735 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:58.562747 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.564173 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.564179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:58.566446 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:58.566482 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.568220 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.568226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:58.568374 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:58.568405 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 02:52:58.568480 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 02:52:58.568578 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10705,"thread_start_us":10977,"threads_started":1}
I20260620 02:52:58.568871 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:58.551259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59141 (local address 127.0.97.1:36377)
0620 02:52:58.551434 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:58.551440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:58.561875 (+ 10435us) server_negotiation.cc:407] Connection header received
0620 02:52:58.562146 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:58.562154 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:58.562231 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:58.562290 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:58.562945 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.564008 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.566692 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.568033 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.568173 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:58.568286 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:58.568547 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 02:52:58.568663 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 02:52:58.569864 8149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "941e1ab5437540d1b27be85551498332" candidate_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b60b7bcf07f449a98435b338b4a952c" is_pre_election: true
I20260620 02:52:58.570986 8220 leader_election.cc:302] T 941e1ab5437540d1b27be85551498332 P 8eeb98eed5c94cc898e55490f1ebd0cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8eeb98eed5c94cc898e55490f1ebd0cb; no voters: 03703e96a41b434ab9cd6c41f16914bb, 6b60b7bcf07f449a98435b338b4a952c
I20260620 02:52:58.571419 8506 raft_consensus.cc:2726] T 941e1ab5437540d1b27be85551498332 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 02:52:58.572153 7989 catalog_manager.cc:5495] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b60b7bcf07f449a98435b338b4a952c (127.0.97.1). New cstate: current_term: 1 leader_uuid: "6b60b7bcf07f449a98435b338b4a952c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } health_report { overall_health: HEALTHY } } }
I20260620 02:52:58.611714 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:58.606490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36414 (local address 127.0.97.1:36377)
0620 02:52:58.606699 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:58.606704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:58.606723 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:52:58.606851 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:58.606854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:58.606949 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:58.607090 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:52:58.607426 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.608247 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.609613 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:58.610037 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:58.610136 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:58.610607 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:52:58.611198 (+ 591us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:52:58.611314 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:58.611398 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 02:52:58.611464 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 02:52:58.620013 8426 raft_consensus.cc:1261] T 941e1ab5437540d1b27be85551498332 P 03703e96a41b434ab9cd6c41f16914bb [term 1 FOLLOWER]: Refusing update from remote peer 6b60b7bcf07f449a98435b338b4a952c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:58.620045 8292 raft_consensus.cc:1261] T 941e1ab5437540d1b27be85551498332 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 1 FOLLOWER]: Refusing update from remote peer 6b60b7bcf07f449a98435b338b4a952c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:58.620939 8513 consensus_queue.cc:1035] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c [LEADER]: Connected to new peer: Peer: permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:52:58.621277 8507 consensus_queue.cc:1035] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:52:58.632681 8516 mvcc.cc:205] Tried to move back new op lower bound from 7298760616418045952 to 7298760616215662592. Current Snapshot: MvccSnapshot[applied={T|T < 7298760616418045952}]
I20260620 02:52:58.635705 8518 mvcc.cc:205] Tried to move back new op lower bound from 7298760616418045952 to 7298760616215662592. Current Snapshot: MvccSnapshot[applied={T|T < 7298760616418045952}]
I20260620 02:52:58.646312 7989 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:52:58.646656 7989 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:52:58.650279 7989 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4e7c66c5c1845aebeeedfe4dc6532f0: Sending DeleteTablet for 3 replicas of tablet 941e1ab5437540d1b27be85551498332
I20260620 02:52:58.651417 8406 tablet_service.cc:1514] Processing DeleteTablet for tablet 941e1ab5437540d1b27be85551498332 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:58 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38360
I20260620 02:52:58.651450 8267 tablet_service.cc:1514] Processing DeleteTablet for tablet 941e1ab5437540d1b27be85551498332 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:58 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42746
I20260620 02:52:58.652486 8127 tablet_service.cc:1514] Processing DeleteTablet for tablet 941e1ab5437540d1b27be85551498332 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:58 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36408
I20260620 02:52:58.652505 7989 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:52:58.653213 7989 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:52:58.654888 8528 tablet_replica.cc:330] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c: stopping tablet replica
I20260620 02:52:58.655314 8528 raft_consensus.cc:2229] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:52:58.655965 8528 raft_consensus.cc:2258] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:58.662892 8526 tablet_replica.cc:330] T 941e1ab5437540d1b27be85551498332 P 03703e96a41b434ab9cd6c41f16914bb: stopping tablet replica
I20260620 02:52:58.663318 8528 ts_tablet_manager.cc:1905] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:58.663329 8526 raft_consensus.cc:2229] T 941e1ab5437540d1b27be85551498332 P 03703e96a41b434ab9cd6c41f16914bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:52:58.663833 8526 raft_consensus.cc:2258] T 941e1ab5437540d1b27be85551498332 P 03703e96a41b434ab9cd6c41f16914bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:58.664810 8266 tablet_service.cc:1467] Processing CreateTablet for tablet 9b23831f10864a34943217e4eb11ab36 (DEFAULT_TABLE table=test-table [id=3b8eff760547423195deb4b3b2d74aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:58.665340 8266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b23831f10864a34943217e4eb11ab36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:58.665225 8405 tablet_service.cc:1467] Processing CreateTablet for tablet 9b23831f10864a34943217e4eb11ab36 (DEFAULT_TABLE table=test-table [id=3b8eff760547423195deb4b3b2d74aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:58.665737 8405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b23831f10864a34943217e4eb11ab36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:58.666308 8127 tablet_service.cc:1467] Processing CreateTablet for tablet 9b23831f10864a34943217e4eb11ab36 (DEFAULT_TABLE table=test-table [id=3b8eff760547423195deb4b3b2d74aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:58.666749 8127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b23831f10864a34943217e4eb11ab36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:58.670348 8528 ts_tablet_manager.cc:1918] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:52:58.670645 8528 log.cc:1196] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-0/wal/wals/941e1ab5437540d1b27be85551498332
I20260620 02:52:58.671157 8528 ts_tablet_manager.cc:1939] T 941e1ab5437540d1b27be85551498332 P 6b60b7bcf07f449a98435b338b4a952c: Deleting consensus metadata
I20260620 02:52:58.671805 8501 tablet_bootstrap.cc:492] T 9b23831f10864a34943217e4eb11ab36 P 8eeb98eed5c94cc898e55490f1ebd0cb: Bootstrap starting.
I20260620 02:52:58.672256 8504 tablet_bootstrap.cc:492] T 9b23831f10864a34943217e4eb11ab36 P 03703e96a41b434ab9cd6c41f16914bb: Bootstrap starting.
I20260620 02:52:58.673411 8501 tablet_bootstrap.cc:654] T 9b23831f10864a34943217e4eb11ab36 P 8eeb98eed5c94cc898e55490f1ebd0cb: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:58.673806 8504 tablet_bootstrap.cc:654] T 9b23831f10864a34943217e4eb11ab36 P 03703e96a41b434ab9cd6c41f16914bb: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:58.676700 8500 tablet_bootstrap.cc:492] T 9b23831f10864a34943217e4eb11ab36 P 6b60b7bcf07f449a98435b338b4a952c: Bootstrap starting.
I20260620 02:52:58.678517 8500 tablet_bootstrap.cc:654] T 9b23831f10864a34943217e4eb11ab36 P 6b60b7bcf07f449a98435b338b4a952c: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:58.679103 8501 tablet_bootstrap.cc:492] T 9b23831f10864a34943217e4eb11ab36 P 8eeb98eed5c94cc898e55490f1ebd0cb: No bootstrap required, opened a new log
I20260620 02:52:58.679232 8501 ts_tablet_manager.cc:1397] T 9b23831f10864a34943217e4eb11ab36 P 8eeb98eed5c94cc898e55490f1ebd0cb: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20260620 02:52:58.680276 8501 raft_consensus.cc:348] T 9b23831f10864a34943217e4eb11ab36 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } }
I20260620 02:52:58.680572 8501 raft_consensus.cc:374] T 9b23831f10864a34943217e4eb11ab36 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:58.680691 8501 raft_consensus.cc:729] T 9b23831f10864a34943217e4eb11ab36 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eeb98eed5c94cc898e55490f1ebd0cb, State: Initialized, Role: FOLLOWER
I20260620 02:52:58.681085 8501 consensus_queue.cc:260] T 9b23831f10864a34943217e4eb11ab36 P 8eeb98eed5c94cc898e55490f1ebd0cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } }
I20260620 02:52:58.681843 8501 ts_tablet_manager.cc:1428] T 9b23831f10864a34943217e4eb11ab36 P 8eeb98eed5c94cc898e55490f1ebd0cb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 02:52:58.686970 7977 catalog_manager.cc:4841] TS 6b60b7bcf07f449a98435b338b4a952c (127.0.97.1:36377): tablet 941e1ab5437540d1b27be85551498332 (table test-table [id=418b2d24fccb44b1834ba9d885e7a980]) successfully deleted
I20260620 02:52:58.688934 8527 tablet_replica.cc:330] T 941e1ab5437540d1b27be85551498332 P 8eeb98eed5c94cc898e55490f1ebd0cb: stopping tablet replica
I20260620 02:52:58.689213 8526 ts_tablet_manager.cc:1905] T 941e1ab5437540d1b27be85551498332 P 03703e96a41b434ab9cd6c41f16914bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:58.689379 8527 raft_consensus.cc:2229] T 941e1ab5437540d1b27be85551498332 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:52:58.689886 8527 raft_consensus.cc:2258] T 941e1ab5437540d1b27be85551498332 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:58.690419 8500 tablet_bootstrap.cc:492] T 9b23831f10864a34943217e4eb11ab36 P 6b60b7bcf07f449a98435b338b4a952c: No bootstrap required, opened a new log
I20260620 02:52:58.690462 8504 tablet_bootstrap.cc:492] T 9b23831f10864a34943217e4eb11ab36 P 03703e96a41b434ab9cd6c41f16914bb: No bootstrap required, opened a new log
I20260620 02:52:58.690568 8500 ts_tablet_manager.cc:1397] T 9b23831f10864a34943217e4eb11ab36 P 6b60b7bcf07f449a98435b338b4a952c: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.007s
I20260620 02:52:58.690734 8504 ts_tablet_manager.cc:1397] T 9b23831f10864a34943217e4eb11ab36 P 03703e96a41b434ab9cd6c41f16914bb: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.009s
I20260620 02:52:58.691490 8500 raft_consensus.cc:348] T 9b23831f10864a34943217e4eb11ab36 P 6b60b7bcf07f449a98435b338b4a952c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } }
I20260620 02:52:58.691715 8500 raft_consensus.cc:374] T 9b23831f10864a34943217e4eb11ab36 P 6b60b7bcf07f449a98435b338b4a952c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:58.691809 8500 raft_consensus.cc:729] T 9b23831f10864a34943217e4eb11ab36 P 6b60b7bcf07f449a98435b338b4a952c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b60b7bcf07f449a98435b338b4a952c, State: Initialized, Role: FOLLOWER
I20260620 02:52:58.691596 8504 raft_consensus.cc:348] T 9b23831f10864a34943217e4eb11ab36 P 03703e96a41b434ab9cd6c41f16914bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } }
I20260620 02:52:58.692026 8527 ts_tablet_manager.cc:1905] T 941e1ab5437540d1b27be85551498332 P 8eeb98eed5c94cc898e55490f1ebd0cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:58.692026 8504 raft_consensus.cc:374] T 9b23831f10864a34943217e4eb11ab36 P 03703e96a41b434ab9cd6c41f16914bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:58.692191 8504 raft_consensus.cc:729] T 9b23831f10864a34943217e4eb11ab36 P 03703e96a41b434ab9cd6c41f16914bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03703e96a41b434ab9cd6c41f16914bb, State: Initialized, Role: FOLLOWER
I20260620 02:52:58.692083 8500 consensus_queue.cc:260] T 9b23831f10864a34943217e4eb11ab36 P 6b60b7bcf07f449a98435b338b4a952c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } }
I20260620 02:52:58.692524 8504 consensus_queue.cc:260] T 9b23831f10864a34943217e4eb11ab36 P 03703e96a41b434ab9cd6c41f16914bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } }
I20260620 02:52:58.692695 8500 ts_tablet_manager.cc:1428] T 9b23831f10864a34943217e4eb11ab36 P 6b60b7bcf07f449a98435b338b4a952c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 02:52:58.693290 8504 ts_tablet_manager.cc:1428] T 9b23831f10864a34943217e4eb11ab36 P 03703e96a41b434ab9cd6c41f16914bb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 02:52:58.698302 8526 ts_tablet_manager.cc:1918] T 941e1ab5437540d1b27be85551498332 P 03703e96a41b434ab9cd6c41f16914bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:52:58.698527 8526 log.cc:1196] T 941e1ab5437540d1b27be85551498332 P 03703e96a41b434ab9cd6c41f16914bb: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-2/wal/wals/941e1ab5437540d1b27be85551498332
I20260620 02:52:58.699033 8526 ts_tablet_manager.cc:1939] T 941e1ab5437540d1b27be85551498332 P 03703e96a41b434ab9cd6c41f16914bb: Deleting consensus metadata
I20260620 02:52:58.700641 8527 ts_tablet_manager.cc:1918] T 941e1ab5437540d1b27be85551498332 P 8eeb98eed5c94cc898e55490f1ebd0cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:52:58.700855 8527 log.cc:1196] T 941e1ab5437540d1b27be85551498332 P 8eeb98eed5c94cc898e55490f1ebd0cb: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-1/wal/wals/941e1ab5437540d1b27be85551498332
I20260620 02:52:58.700842 7977 catalog_manager.cc:4841] TS 03703e96a41b434ab9cd6c41f16914bb (127.0.97.3:35929): tablet 941e1ab5437540d1b27be85551498332 (table test-table [id=418b2d24fccb44b1834ba9d885e7a980]) successfully deleted
I20260620 02:52:58.701519 8527 ts_tablet_manager.cc:1939] T 941e1ab5437540d1b27be85551498332 P 8eeb98eed5c94cc898e55490f1ebd0cb: Deleting consensus metadata
I20260620 02:52:58.702984 7976 catalog_manager.cc:4841] TS 8eeb98eed5c94cc898e55490f1ebd0cb (127.0.97.2:40485): tablet 941e1ab5437540d1b27be85551498332 (table test-table [id=418b2d24fccb44b1834ba9d885e7a980]) successfully deleted
I20260620 02:52:58.960130 8507 raft_consensus.cc:482] T 9b23831f10864a34943217e4eb11ab36 P 6b60b7bcf07f449a98435b338b4a952c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:52:58.960500 8507 raft_consensus.cc:504] T 9b23831f10864a34943217e4eb11ab36 P 6b60b7bcf07f449a98435b338b4a952c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } }
I20260620 02:52:58.961328 8507 leader_election.cc:288] T 9b23831f10864a34943217e4eb11ab36 P 6b60b7bcf07f449a98435b338b4a952c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8eeb98eed5c94cc898e55490f1ebd0cb (127.0.97.2:40485), 03703e96a41b434ab9cd6c41f16914bb (127.0.97.3:35929)
I20260620 02:52:58.961962 8292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b23831f10864a34943217e4eb11ab36" candidate_uuid: "6b60b7bcf07f449a98435b338b4a952c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" is_pre_election: true
I20260620 02:52:58.961982 8426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b23831f10864a34943217e4eb11ab36" candidate_uuid: "6b60b7bcf07f449a98435b338b4a952c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03703e96a41b434ab9cd6c41f16914bb" is_pre_election: true
I20260620 02:52:58.962270 8426 raft_consensus.cc:2454] T 9b23831f10864a34943217e4eb11ab36 P 03703e96a41b434ab9cd6c41f16914bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b60b7bcf07f449a98435b338b4a952c in term 0.
I20260620 02:52:58.962280 8292 raft_consensus.cc:2454] T 9b23831f10864a34943217e4eb11ab36 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b60b7bcf07f449a98435b338b4a952c in term 0.
I20260620 02:52:58.962949 8079 leader_election.cc:302] T 9b23831f10864a34943217e4eb11ab36 P 6b60b7bcf07f449a98435b338b4a952c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 03703e96a41b434ab9cd6c41f16914bb, 6b60b7bcf07f449a98435b338b4a952c, 8eeb98eed5c94cc898e55490f1ebd0cb; no voters:
I20260620 02:52:58.963327 8507 raft_consensus.cc:2781] T 9b23831f10864a34943217e4eb11ab36 P 6b60b7bcf07f449a98435b338b4a952c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:52:58.963461 8507 raft_consensus.cc:482] T 9b23831f10864a34943217e4eb11ab36 P 6b60b7bcf07f449a98435b338b4a952c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:52:58.963619 8507 raft_consensus.cc:3037] T 9b23831f10864a34943217e4eb11ab36 P 6b60b7bcf07f449a98435b338b4a952c [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:58.967224 8507 raft_consensus.cc:504] T 9b23831f10864a34943217e4eb11ab36 P 6b60b7bcf07f449a98435b338b4a952c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } }
I20260620 02:52:58.968039 8507 leader_election.cc:288] T 9b23831f10864a34943217e4eb11ab36 P 6b60b7bcf07f449a98435b338b4a952c [CANDIDATE]: Term 1 election: Requested vote from peers 8eeb98eed5c94cc898e55490f1ebd0cb (127.0.97.2:40485), 03703e96a41b434ab9cd6c41f16914bb (127.0.97.3:35929)
I20260620 02:52:58.968637 8292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b23831f10864a34943217e4eb11ab36" candidate_uuid: "6b60b7bcf07f449a98435b338b4a952c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb"
I20260620 02:52:58.968670 8426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b23831f10864a34943217e4eb11ab36" candidate_uuid: "6b60b7bcf07f449a98435b338b4a952c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03703e96a41b434ab9cd6c41f16914bb"
I20260620 02:52:58.968873 8292 raft_consensus.cc:3037] T 9b23831f10864a34943217e4eb11ab36 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:58.968899 8426 raft_consensus.cc:3037] T 9b23831f10864a34943217e4eb11ab36 P 03703e96a41b434ab9cd6c41f16914bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:58.971012 8292 raft_consensus.cc:2454] T 9b23831f10864a34943217e4eb11ab36 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b60b7bcf07f449a98435b338b4a952c in term 1.
I20260620 02:52:58.971509 8079 leader_election.cc:302] T 9b23831f10864a34943217e4eb11ab36 P 6b60b7bcf07f449a98435b338b4a952c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b60b7bcf07f449a98435b338b4a952c, 8eeb98eed5c94cc898e55490f1ebd0cb; no voters:
I20260620 02:52:58.971860 8507 raft_consensus.cc:2781] T 9b23831f10864a34943217e4eb11ab36 P 6b60b7bcf07f449a98435b338b4a952c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:58.971989 8426 raft_consensus.cc:2454] T 9b23831f10864a34943217e4eb11ab36 P 03703e96a41b434ab9cd6c41f16914bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b60b7bcf07f449a98435b338b4a952c in term 1.
I20260620 02:52:58.972046 8507 raft_consensus.cc:686] T 9b23831f10864a34943217e4eb11ab36 P 6b60b7bcf07f449a98435b338b4a952c [term 1 LEADER]: Becoming Leader. State: Replica: 6b60b7bcf07f449a98435b338b4a952c, State: Running, Role: LEADER
I20260620 02:52:58.972637 8507 consensus_queue.cc:237] T 9b23831f10864a34943217e4eb11ab36 P 6b60b7bcf07f449a98435b338b4a952c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } }
I20260620 02:52:58.975822 7990 catalog_manager.cc:5495] T 9b23831f10864a34943217e4eb11ab36 P 6b60b7bcf07f449a98435b338b4a952c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b60b7bcf07f449a98435b338b4a952c (127.0.97.1). New cstate: current_term: 1 leader_uuid: "6b60b7bcf07f449a98435b338b4a952c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } health_report { overall_health: UNKNOWN } } }
Jun 20 02:52:59 dist-test-slave-8p8c krb5kdc[7945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923976, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:52:59.079689 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:59.061115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45506 (local address 127.0.97.62:39285)
0620 02:52:59.062160 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:59.062167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:59.062194 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:52:59.062235 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:59.062239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:59.062292 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:59.062445 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:59.063530 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.064403 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.065736 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.066353 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.066600 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:59.074264 (+ 7664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:59.074268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:59.074271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:59.074299 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:59.076955 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:59.077636 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:59.077644 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:59.077647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:59.077759 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:59.078161 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:59.078166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:59.078169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:59.078495 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:59.078645 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:59.079050 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 02:52:59.079252 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":898,"thread_start_us":723,"threads_started":1}
I20260620 02:52:59.079748 8538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:59.061062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39285 (local address 127.0.0.1:45506)
0620 02:52:59.061826 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:59.061853 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:59.062035 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:59.062461 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:59.062466 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:59.063192 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:52:59.063372 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:59.063381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.064562 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.064569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:59.065584 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:59.065596 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.066630 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.066637 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:59.066826 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:59.071779 (+ 4953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:52:59.071799 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:52:59.074049 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:52:59.077112 (+ 3063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:59.077124 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:59.077133 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:59.077515 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:59.077869 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:52:59.077874 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:52:59.077877 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:52:59.078036 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:52:59.078634 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:52:59.078642 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:52:59.078804 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 02:52:59.078967 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 02:52:59.079376 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":135,"thread_start_us":392,"threads_started":1}
I20260620 02:52:59.085098 7990 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45506:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 02:52:59.086321 7990 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:52:59.093395 8127 tablet_service.cc:1467] Processing CreateTablet for tablet 8a0a8453266f4edeb7db67a1db120b75 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f4189e05355742fe877b0c61d1e720fb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:52:59.093417 8405 tablet_service.cc:1467] Processing CreateTablet for tablet 8a0a8453266f4edeb7db67a1db120b75 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f4189e05355742fe877b0c61d1e720fb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:52:59.093971 8405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a0a8453266f4edeb7db67a1db120b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:59.094003 8127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a0a8453266f4edeb7db67a1db120b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:59.099395 8504 tablet_bootstrap.cc:492] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb: Bootstrap starting.
I20260620 02:52:59.099836 8267 tablet_service.cc:1467] Processing CreateTablet for tablet 8a0a8453266f4edeb7db67a1db120b75 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f4189e05355742fe877b0c61d1e720fb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 02:52:59.100132 8500 tablet_bootstrap.cc:492] T 8a0a8453266f4edeb7db67a1db120b75 P 6b60b7bcf07f449a98435b338b4a952c: Bootstrap starting.
I20260620 02:52:59.100400 8267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a0a8453266f4edeb7db67a1db120b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:59.101044 8504 tablet_bootstrap.cc:654] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:59.101696 8500 tablet_bootstrap.cc:654] T 8a0a8453266f4edeb7db67a1db120b75 P 6b60b7bcf07f449a98435b338b4a952c: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:59.102999 8504 tablet_bootstrap.cc:492] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb: No bootstrap required, opened a new log
I20260620 02:52:59.103169 8504 ts_tablet_manager.cc:1397] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 02:52:59.103777 8500 tablet_bootstrap.cc:492] T 8a0a8453266f4edeb7db67a1db120b75 P 6b60b7bcf07f449a98435b338b4a952c: No bootstrap required, opened a new log
I20260620 02:52:59.103930 8500 ts_tablet_manager.cc:1397] T 8a0a8453266f4edeb7db67a1db120b75 P 6b60b7bcf07f449a98435b338b4a952c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260620 02:52:59.103900 8504 raft_consensus.cc:348] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } }
I20260620 02:52:59.104175 8504 raft_consensus.cc:374] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:59.104279 8504 raft_consensus.cc:729] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03703e96a41b434ab9cd6c41f16914bb, State: Initialized, Role: FOLLOWER
I20260620 02:52:59.104547 8504 consensus_queue.cc:260] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } }
I20260620 02:52:59.104578 8500 raft_consensus.cc:348] T 8a0a8453266f4edeb7db67a1db120b75 P 6b60b7bcf07f449a98435b338b4a952c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } }
I20260620 02:52:59.104816 8500 raft_consensus.cc:374] T 8a0a8453266f4edeb7db67a1db120b75 P 6b60b7bcf07f449a98435b338b4a952c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:59.104890 8500 raft_consensus.cc:729] T 8a0a8453266f4edeb7db67a1db120b75 P 6b60b7bcf07f449a98435b338b4a952c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b60b7bcf07f449a98435b338b4a952c, State: Initialized, Role: FOLLOWER
I20260620 02:52:59.104948 8508 tablet_replica.cc:439] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } }
I20260620 02:52:59.105233 8508 tablet_replica.cc:442] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:52:59.105109 8500 consensus_queue.cc:260] T 8a0a8453266f4edeb7db67a1db120b75 P 6b60b7bcf07f449a98435b338b4a952c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } }
I20260620 02:52:59.105532 8507 tablet_replica.cc:439] T 8a0a8453266f4edeb7db67a1db120b75 P 6b60b7bcf07f449a98435b338b4a952c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } }
I20260620 02:52:59.105851 8507 tablet_replica.cc:442] T 8a0a8453266f4edeb7db67a1db120b75 P 6b60b7bcf07f449a98435b338b4a952c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:52:59.106160 8500 ts_tablet_manager.cc:1428] T 8a0a8453266f4edeb7db67a1db120b75 P 6b60b7bcf07f449a98435b338b4a952c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260620 02:52:59.107043 8504 ts_tablet_manager.cc:1428] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 02:52:59.108516 8501 tablet_bootstrap.cc:492] T 8a0a8453266f4edeb7db67a1db120b75 P 8eeb98eed5c94cc898e55490f1ebd0cb: Bootstrap starting.
I20260620 02:52:59.110210 8501 tablet_bootstrap.cc:654] T 8a0a8453266f4edeb7db67a1db120b75 P 8eeb98eed5c94cc898e55490f1ebd0cb: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:59.112027 8501 tablet_bootstrap.cc:492] T 8a0a8453266f4edeb7db67a1db120b75 P 8eeb98eed5c94cc898e55490f1ebd0cb: No bootstrap required, opened a new log
I20260620 02:52:59.112200 8501 ts_tablet_manager.cc:1397] T 8a0a8453266f4edeb7db67a1db120b75 P 8eeb98eed5c94cc898e55490f1ebd0cb: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 02:52:59.112766 8501 raft_consensus.cc:348] T 8a0a8453266f4edeb7db67a1db120b75 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } }
I20260620 02:52:59.112982 8501 raft_consensus.cc:374] T 8a0a8453266f4edeb7db67a1db120b75 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:59.113065 8501 raft_consensus.cc:729] T 8a0a8453266f4edeb7db67a1db120b75 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eeb98eed5c94cc898e55490f1ebd0cb, State: Initialized, Role: FOLLOWER
I20260620 02:52:59.113296 8501 consensus_queue.cc:260] T 8a0a8453266f4edeb7db67a1db120b75 P 8eeb98eed5c94cc898e55490f1ebd0cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } }
I20260620 02:52:59.113822 8501 ts_tablet_manager.cc:1428] T 8a0a8453266f4edeb7db67a1db120b75 P 8eeb98eed5c94cc898e55490f1ebd0cb: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 02:52:59.113684 8506 tablet_replica.cc:439] T 8a0a8453266f4edeb7db67a1db120b75 P 8eeb98eed5c94cc898e55490f1ebd0cb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } }
I20260620 02:52:59.113967 8506 tablet_replica.cc:442] T 8a0a8453266f4edeb7db67a1db120b75 P 8eeb98eed5c94cc898e55490f1ebd0cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:52:59.303365 8508 raft_consensus.cc:482] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:52:59.303568 8508 raft_consensus.cc:504] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } }
I20260620 02:52:59.305074 8508 leader_election.cc:288] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b60b7bcf07f449a98435b338b4a952c (127.0.97.1:36377), 8eeb98eed5c94cc898e55490f1ebd0cb (127.0.97.2:40485)
I20260620 02:52:59.312239 8544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:59.305393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36377 (local address 127.0.97.3:36003)
0620 02:52:59.305979 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:59.306006 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:59.306188 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:59.307318 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:59.307323 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:59.307348 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:59.307651 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:59.307660 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.308749 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.308754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:59.310198 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:59.310209 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.311406 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.311411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:59.311602 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:59.311650 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 02:52:59.311740 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 02:52:59.311835 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":4,"thread_start_us":1066,"threads_started":1}
I20260620 02:52:59.312245 8545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:59.305503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36003 (local address 127.0.97.1:36377)
0620 02:52:59.307022 (+ 1519us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:59.307029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:59.307051 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:52:59.307125 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:59.307129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:59.307186 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:59.307268 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:59.307823 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.308586 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.310391 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.311230 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.311328 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:59.311399 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:59.311815 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 02:52:59.311972 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1382,"thread_start_us":1253,"threads_started":1}
I20260620 02:52:59.312395 8546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:59.305731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36999 (local address 127.0.97.2:40485)
0620 02:52:59.307154 (+ 1423us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:59.307161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:59.307176 (+ 15us) server_negotiation.cc:407] Connection header received
0620 02:52:59.307230 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:59.307233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:59.307327 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:59.307436 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:52:59.308079 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.308894 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.310648 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.311418 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.311541 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:59.311637 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:59.311947 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 02:52:59.312090 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1330,"thread_start_us":1225,"threads_started":1}
I20260620 02:52:59.312333 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:59.305420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40485 (local address 127.0.97.3:36999)
0620 02:52:59.306776 (+ 1356us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:59.306792 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:59.306890 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:59.307465 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:59.307469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:59.307493 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:52:59.307827 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:59.307837 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.309055 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.309059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:59.310511 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:59.310526 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.311603 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.311607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:59.311781 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:59.311813 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 02:52:59.311908 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 02:52:59.312033 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1215,"thread_start_us":494,"threads_started":1}
I20260620 02:52:59.313125 8149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0a8453266f4edeb7db67a1db120b75" candidate_uuid: "03703e96a41b434ab9cd6c41f16914bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b60b7bcf07f449a98435b338b4a952c" is_pre_election: true
I20260620 02:52:59.313161 8292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0a8453266f4edeb7db67a1db120b75" candidate_uuid: "03703e96a41b434ab9cd6c41f16914bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" is_pre_election: true
I20260620 02:52:59.313386 8149 raft_consensus.cc:2454] T 8a0a8453266f4edeb7db67a1db120b75 P 6b60b7bcf07f449a98435b338b4a952c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03703e96a41b434ab9cd6c41f16914bb in term 0.
I20260620 02:52:59.313390 8292 raft_consensus.cc:2454] T 8a0a8453266f4edeb7db67a1db120b75 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03703e96a41b434ab9cd6c41f16914bb in term 0.
I20260620 02:52:59.313858 8360 leader_election.cc:302] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03703e96a41b434ab9cd6c41f16914bb, 6b60b7bcf07f449a98435b338b4a952c; no voters:
I20260620 02:52:59.314253 8508 raft_consensus.cc:2781] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:52:59.314405 8508 raft_consensus.cc:482] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:52:59.314502 8508 raft_consensus.cc:3037] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:59.316491 8508 raft_consensus.cc:504] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } }
I20260620 02:52:59.317046 8508 leader_election.cc:288] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb [CANDIDATE]: Term 1 election: Requested vote from peers 6b60b7bcf07f449a98435b338b4a952c (127.0.97.1:36377), 8eeb98eed5c94cc898e55490f1ebd0cb (127.0.97.2:40485)
I20260620 02:52:59.317477 8149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0a8453266f4edeb7db67a1db120b75" candidate_uuid: "03703e96a41b434ab9cd6c41f16914bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b60b7bcf07f449a98435b338b4a952c"
I20260620 02:52:59.317530 8292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0a8453266f4edeb7db67a1db120b75" candidate_uuid: "03703e96a41b434ab9cd6c41f16914bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb"
I20260620 02:52:59.317726 8149 raft_consensus.cc:3037] T 8a0a8453266f4edeb7db67a1db120b75 P 6b60b7bcf07f449a98435b338b4a952c [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:59.317766 8292 raft_consensus.cc:3037] T 8a0a8453266f4edeb7db67a1db120b75 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:59.319813 8149 raft_consensus.cc:2454] T 8a0a8453266f4edeb7db67a1db120b75 P 6b60b7bcf07f449a98435b338b4a952c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03703e96a41b434ab9cd6c41f16914bb in term 1.
I20260620 02:52:59.319932 8292 raft_consensus.cc:2454] T 8a0a8453266f4edeb7db67a1db120b75 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03703e96a41b434ab9cd6c41f16914bb in term 1.
I20260620 02:52:59.320194 8360 leader_election.cc:302] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03703e96a41b434ab9cd6c41f16914bb, 6b60b7bcf07f449a98435b338b4a952c; no voters:
I20260620 02:52:59.320480 8508 raft_consensus.cc:2781] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:59.320809 8508 raft_consensus.cc:686] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb [term 1 LEADER]: Becoming Leader. State: Replica: 03703e96a41b434ab9cd6c41f16914bb, State: Running, Role: LEADER
I20260620 02:52:59.321159 8508 consensus_queue.cc:237] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } }
I20260620 02:52:59.322953 8547 tablet_replica.cc:439] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb: TxnStatusTablet state changed. Reason: New leader 03703e96a41b434ab9cd6c41f16914bb. Latest consensus state: current_term: 1 leader_uuid: "03703e96a41b434ab9cd6c41f16914bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } }
I20260620 02:52:59.323362 8547 tablet_replica.cc:442] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:52:59.324110 8549 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 02:52:59.324328 8549 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 02:52:59.324795 7989 catalog_manager.cc:5495] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 03703e96a41b434ab9cd6c41f16914bb (127.0.97.3). New cstate: current_term: 1 leader_uuid: "03703e96a41b434ab9cd6c41f16914bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } health_report { overall_health: UNKNOWN } } }
I20260620 02:52:59.346879 8538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:59.341021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35929 (local address 127.0.0.1:38374)
0620 02:52:59.341236 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:59.341272 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:59.341417 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:59.342010 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:59.342015 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:59.342028 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:52:59.342233 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:59.342246 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.343401 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.343409 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:59.344390 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:59.344400 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.345188 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.345193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:59.345366 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:59.345414 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:52:59.346271 (+ 857us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:52:59.346289 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 02:52:59.346419 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 02:52:59.346563 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 02:52:59.346909 8552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:59.341115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38374 (local address 127.0.97.3:35929)
0620 02:52:59.341674 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:59.341678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:59.341697 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:52:59.341766 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:59.341770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:59.341835 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:59.341994 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:52:59.342451 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.343276 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.344523 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.344997 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.345157 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:59.345631 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:52:59.346165 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:52:59.346319 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:59.346428 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 02:52:59.346507 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":248,"threads_started":1}
I20260620 02:52:59.353140 8292 raft_consensus.cc:1261] T 8a0a8453266f4edeb7db67a1db120b75 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 1 FOLLOWER]: Refusing update from remote peer 03703e96a41b434ab9cd6c41f16914bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:59.353286 8149 raft_consensus.cc:1261] T 8a0a8453266f4edeb7db67a1db120b75 P 6b60b7bcf07f449a98435b338b4a952c [term 1 FOLLOWER]: Refusing update from remote peer 03703e96a41b434ab9cd6c41f16914bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:59.353916 8508 consensus_queue.cc:1035] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:52:59.354230 8547 consensus_queue.cc:1035] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:52:59.362239 8507 tablet_replica.cc:439] T 8a0a8453266f4edeb7db67a1db120b75 P 6b60b7bcf07f449a98435b338b4a952c: TxnStatusTablet state changed. Reason: New leader 03703e96a41b434ab9cd6c41f16914bb. Latest consensus state: current_term: 1 leader_uuid: "03703e96a41b434ab9cd6c41f16914bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } }
I20260620 02:52:59.362540 8507 tablet_replica.cc:442] T 8a0a8453266f4edeb7db67a1db120b75 P 6b60b7bcf07f449a98435b338b4a952c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:52:59.363866 8506 tablet_replica.cc:439] T 8a0a8453266f4edeb7db67a1db120b75 P 8eeb98eed5c94cc898e55490f1ebd0cb: TxnStatusTablet state changed. Reason: New leader 03703e96a41b434ab9cd6c41f16914bb. Latest consensus state: current_term: 1 leader_uuid: "03703e96a41b434ab9cd6c41f16914bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } }
I20260620 02:52:59.364156 8506 tablet_replica.cc:442] T 8a0a8453266f4edeb7db67a1db120b75 P 8eeb98eed5c94cc898e55490f1ebd0cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:52:59.366464 8547 tablet_replica.cc:439] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03703e96a41b434ab9cd6c41f16914bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } }
I20260620 02:52:59.366426 8558 tablet_replica.cc:439] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03703e96a41b434ab9cd6c41f16914bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } }
I20260620 02:52:59.366859 8558 tablet_replica.cc:442] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:52:59.366835 8547 tablet_replica.cc:442] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb: This TxnStatusTablet replica's current role is: LEADER
I20260620 02:52:59.379230 8507 tablet_replica.cc:439] T 8a0a8453266f4edeb7db67a1db120b75 P 6b60b7bcf07f449a98435b338b4a952c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03703e96a41b434ab9cd6c41f16914bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } }
I20260620 02:52:59.379577 8507 tablet_replica.cc:442] T 8a0a8453266f4edeb7db67a1db120b75 P 6b60b7bcf07f449a98435b338b4a952c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:52:59.386648 8506 tablet_replica.cc:439] T 8a0a8453266f4edeb7db67a1db120b75 P 8eeb98eed5c94cc898e55490f1ebd0cb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03703e96a41b434ab9cd6c41f16914bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } }
I20260620 02:52:59.386955 8506 tablet_replica.cc:442] T 8a0a8453266f4edeb7db67a1db120b75 P 8eeb98eed5c94cc898e55490f1ebd0cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 02:52:59.394161 8552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:59.387806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59409 (local address 127.0.97.3:35929)
0620 02:52:59.387967 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:59.387973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:59.388537 (+ 564us) server_negotiation.cc:407] Connection header received
0620 02:52:59.388759 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:59.388765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:59.388845 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:59.388910 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:52:59.389579 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.390439 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.392406 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.392946 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.393038 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:59.393556 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:52:59.393653 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:52:59.393722 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:59.393868 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 02:52:59.393925 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 02:52:59.394496 8565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:59.387550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35929 (local address 127.0.97.1:59409)
0620 02:52:59.388413 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:59.388432 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:59.388589 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:59.389116 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:59.389120 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:59.389132 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:52:59.389419 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:59.389431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.390569 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.390574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:59.392276 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:59.392292 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.393093 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.393100 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:59.393307 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:59.393385 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:52:59.393749 (+ 364us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:52:59.393761 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 02:52:59.393847 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 02:52:59.393985 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":572,"threads_started":1}
I20260620 02:52:59.402765 8426 raft_consensus.cc:1261] T 9b23831f10864a34943217e4eb11ab36 P 03703e96a41b434ab9cd6c41f16914bb [term 1 FOLLOWER]: Refusing update from remote peer 6b60b7bcf07f449a98435b338b4a952c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:59.402796 8292 raft_consensus.cc:1261] T 9b23831f10864a34943217e4eb11ab36 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 1 FOLLOWER]: Refusing update from remote peer 6b60b7bcf07f449a98435b338b4a952c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:59.403283 8521 consensus_queue.cc:1035] T 9b23831f10864a34943217e4eb11ab36 P 6b60b7bcf07f449a98435b338b4a952c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:52:59.403685 8507 consensus_queue.cc:1035] T 9b23831f10864a34943217e4eb11ab36 P 6b60b7bcf07f449a98435b338b4a952c [LEADER]: Connected to new peer: Peer: permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:52:59.408792 8554 mvcc.cc:205] Tried to move back new op lower bound from 7298760619629207552 to 7298760617874489344. Current Snapshot: MvccSnapshot[applied={T|T < 7298760619629207552}]
I20260620 02:52:59.448210 8545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:59.432352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33991 (local address 127.0.97.1:36377)
0620 02:52:59.432548 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:59.432555 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:59.442564 (+ 10009us) server_negotiation.cc:407] Connection header received
0620 02:52:59.442703 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:59.442709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:59.442802 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:59.442899 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:52:59.443368 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.444452 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.446183 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.446917 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.447043 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:59.447526 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:52:59.447651 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:52:59.447740 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:59.447868 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 02:52:59.447948 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 02:52:59.448231 8571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:59.431057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36377 (local address 127.0.97.3:33991)
0620 02:52:59.442448 (+ 11391us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:59.442469 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:59.442596 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:59.442904 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:59.442906 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:59.442915 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:52:59.443213 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:59.443221 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.444588 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.444593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:59.446045 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:59.446057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.447127 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.447132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:59.447283 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:59.447339 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:52:59.447748 (+ 409us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:52:59.447762 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 02:52:59.447825 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 02:52:59.447924 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11159,"thread_start_us":11606,"threads_started":1}
I20260620 02:52:59.477161 7988 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:52:59.477403 7988 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:52:59.480314 7988 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4e7c66c5c1845aebeeedfe4dc6532f0: Sending DeleteTablet for 3 replicas of tablet 9b23831f10864a34943217e4eb11ab36
I20260620 02:52:59.481082 8127 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b23831f10864a34943217e4eb11ab36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:59 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36408
I20260620 02:52:59.481294 8405 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b23831f10864a34943217e4eb11ab36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:59 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38360
I20260620 02:52:59.481506 8267 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b23831f10864a34943217e4eb11ab36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:59 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42746
I20260620 02:52:59.481730 8573 tablet_replica.cc:330] T 9b23831f10864a34943217e4eb11ab36 P 6b60b7bcf07f449a98435b338b4a952c: stopping tablet replica
I20260620 02:52:59.481925 8573 raft_consensus.cc:2229] T 9b23831f10864a34943217e4eb11ab36 P 6b60b7bcf07f449a98435b338b4a952c [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:52:59.482334 8573 raft_consensus.cc:2258] T 9b23831f10864a34943217e4eb11ab36 P 6b60b7bcf07f449a98435b338b4a952c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:59.485999 8574 tablet_replica.cc:330] T 9b23831f10864a34943217e4eb11ab36 P 03703e96a41b434ab9cd6c41f16914bb: stopping tablet replica
I20260620 02:52:59.486202 8574 raft_consensus.cc:2229] T 9b23831f10864a34943217e4eb11ab36 P 03703e96a41b434ab9cd6c41f16914bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:52:59.486543 8574 raft_consensus.cc:2258] T 9b23831f10864a34943217e4eb11ab36 P 03703e96a41b434ab9cd6c41f16914bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:59.487004 8573 ts_tablet_manager.cc:1905] T 9b23831f10864a34943217e4eb11ab36 P 6b60b7bcf07f449a98435b338b4a952c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:59.487573 8574 ts_tablet_manager.cc:1905] T 9b23831f10864a34943217e4eb11ab36 P 03703e96a41b434ab9cd6c41f16914bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:59.490089 8575 tablet_replica.cc:330] T 9b23831f10864a34943217e4eb11ab36 P 8eeb98eed5c94cc898e55490f1ebd0cb: stopping tablet replica
I20260620 02:52:59.490288 8575 raft_consensus.cc:2229] T 9b23831f10864a34943217e4eb11ab36 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:52:59.490550 8575 raft_consensus.cc:2258] T 9b23831f10864a34943217e4eb11ab36 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:59.492394 8575 ts_tablet_manager.cc:1905] T 9b23831f10864a34943217e4eb11ab36 P 8eeb98eed5c94cc898e55490f1ebd0cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:59.496170 8573 ts_tablet_manager.cc:1918] T 9b23831f10864a34943217e4eb11ab36 P 6b60b7bcf07f449a98435b338b4a952c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:52:59.496351 8573 log.cc:1196] T 9b23831f10864a34943217e4eb11ab36 P 6b60b7bcf07f449a98435b338b4a952c: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-0/wal/wals/9b23831f10864a34943217e4eb11ab36
I20260620 02:52:59.496913 8573 ts_tablet_manager.cc:1939] T 9b23831f10864a34943217e4eb11ab36 P 6b60b7bcf07f449a98435b338b4a952c: Deleting consensus metadata
I20260620 02:52:59.498095 7977 catalog_manager.cc:4841] TS 6b60b7bcf07f449a98435b338b4a952c (127.0.97.1:36377): tablet 9b23831f10864a34943217e4eb11ab36 (table test-table [id=3b8eff760547423195deb4b3b2d74aa4]) successfully deleted
I20260620 02:52:59.499037 8575 ts_tablet_manager.cc:1918] T 9b23831f10864a34943217e4eb11ab36 P 8eeb98eed5c94cc898e55490f1ebd0cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:52:59.499171 8575 log.cc:1196] T 9b23831f10864a34943217e4eb11ab36 P 8eeb98eed5c94cc898e55490f1ebd0cb: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-1/wal/wals/9b23831f10864a34943217e4eb11ab36
I20260620 02:52:59.499529 8575 ts_tablet_manager.cc:1939] T 9b23831f10864a34943217e4eb11ab36 P 8eeb98eed5c94cc898e55490f1ebd0cb: Deleting consensus metadata
I20260620 02:52:59.500415 7976 catalog_manager.cc:4841] TS 8eeb98eed5c94cc898e55490f1ebd0cb (127.0.97.2:40485): tablet 9b23831f10864a34943217e4eb11ab36 (table test-table [id=3b8eff760547423195deb4b3b2d74aa4]) successfully deleted
I20260620 02:52:59.503604 8574 ts_tablet_manager.cc:1918] T 9b23831f10864a34943217e4eb11ab36 P 03703e96a41b434ab9cd6c41f16914bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:52:59.503777 8574 log.cc:1196] T 9b23831f10864a34943217e4eb11ab36 P 03703e96a41b434ab9cd6c41f16914bb: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-2/wal/wals/9b23831f10864a34943217e4eb11ab36
I20260620 02:52:59.504185 8574 ts_tablet_manager.cc:1939] T 9b23831f10864a34943217e4eb11ab36 P 03703e96a41b434ab9cd6c41f16914bb: Deleting consensus metadata
I20260620 02:52:59.505092 7977 catalog_manager.cc:4841] TS 03703e96a41b434ab9cd6c41f16914bb (127.0.97.3:35929): tablet 9b23831f10864a34943217e4eb11ab36 (table test-table [id=3b8eff760547423195deb4b3b2d74aa4]) successfully deleted
Jun 20 02:52:59 dist-test-slave-8p8c krb5kdc[7945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 02:52:59 dist-test-slave-8p8c krb5kdc[7945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:52:59 dist-test-slave-8p8c krb5kdc[7945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 02:52:59.550005 8539 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:52:59.540082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45510 (local address 127.0.97.62:39285)
0620 02:52:59.540285 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:59.540292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:59.540707 (+ 415us) server_negotiation.cc:407] Connection header received
0620 02:52:59.540871 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:59.540877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:59.540973 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:59.541113 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:59.542434 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.543206 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.544410 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.544853 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.544984 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:59.549763 (+ 4779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45510: BlockingRecv error: recv got EOF from 127.0.0.1:45510 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 02:52:59 dist-test-slave-8p8c krb5kdc[7945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 02:52:59.568148 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:59.553982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45520 (local address 127.0.97.62:39285)
0620 02:52:59.554153 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:59.554159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:59.554831 (+ 672us) server_negotiation.cc:407] Connection header received
0620 02:52:59.555043 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:59.555049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:59.555122 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:59.555222 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:52:59.556356 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.557099 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.558308 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.558830 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.558958 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:59.562710 (+ 3752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:52:59.562715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:52:59.562718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:52:59.562751 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:52:59.565455 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:59.566057 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:59.566064 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:59.566068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:59.566189 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:52:59.566662 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:52:59.566668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:52:59.566671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:52:59.566954 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:52:59.567230 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:59.567531 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 02:52:59.567745 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 02:52:59.571497 7988 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:52:59.572146 7988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:52:59.579196 8127 tablet_service.cc:1467] Processing CreateTablet for tablet ac28b4c3b78445efba3d63245d3f0fe9 (DEFAULT_TABLE table=test-table [id=4029f623d9a94599ba240725c7d12fd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:59.579432 8405 tablet_service.cc:1467] Processing CreateTablet for tablet ac28b4c3b78445efba3d63245d3f0fe9 (DEFAULT_TABLE table=test-table [id=4029f623d9a94599ba240725c7d12fd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:59.579679 8127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac28b4c3b78445efba3d63245d3f0fe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:59.579913 8405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac28b4c3b78445efba3d63245d3f0fe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:59.584944 8504 tablet_bootstrap.cc:492] T ac28b4c3b78445efba3d63245d3f0fe9 P 03703e96a41b434ab9cd6c41f16914bb: Bootstrap starting.
I20260620 02:52:59.586481 8504 tablet_bootstrap.cc:654] T ac28b4c3b78445efba3d63245d3f0fe9 P 03703e96a41b434ab9cd6c41f16914bb: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:59.587586 8267 tablet_service.cc:1467] Processing CreateTablet for tablet ac28b4c3b78445efba3d63245d3f0fe9 (DEFAULT_TABLE table=test-table [id=4029f623d9a94599ba240725c7d12fd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:59.588063 8267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac28b4c3b78445efba3d63245d3f0fe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:59.594924 8501 tablet_bootstrap.cc:492] T ac28b4c3b78445efba3d63245d3f0fe9 P 8eeb98eed5c94cc898e55490f1ebd0cb: Bootstrap starting.
I20260620 02:52:59.596524 8501 tablet_bootstrap.cc:654] T ac28b4c3b78445efba3d63245d3f0fe9 P 8eeb98eed5c94cc898e55490f1ebd0cb: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:59.596769 8500 tablet_bootstrap.cc:492] T ac28b4c3b78445efba3d63245d3f0fe9 P 6b60b7bcf07f449a98435b338b4a952c: Bootstrap starting.
I20260620 02:52:59.597353 8504 tablet_bootstrap.cc:492] T ac28b4c3b78445efba3d63245d3f0fe9 P 03703e96a41b434ab9cd6c41f16914bb: No bootstrap required, opened a new log
I20260620 02:52:59.597929 8504 ts_tablet_manager.cc:1397] T ac28b4c3b78445efba3d63245d3f0fe9 P 03703e96a41b434ab9cd6c41f16914bb: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.011s
I20260620 02:52:59.598448 8500 tablet_bootstrap.cc:654] T ac28b4c3b78445efba3d63245d3f0fe9 P 6b60b7bcf07f449a98435b338b4a952c: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:59.599081 8504 raft_consensus.cc:348] T ac28b4c3b78445efba3d63245d3f0fe9 P 03703e96a41b434ab9cd6c41f16914bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } }
I20260620 02:52:59.599704 8504 raft_consensus.cc:374] T ac28b4c3b78445efba3d63245d3f0fe9 P 03703e96a41b434ab9cd6c41f16914bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:59.599893 8504 raft_consensus.cc:729] T ac28b4c3b78445efba3d63245d3f0fe9 P 03703e96a41b434ab9cd6c41f16914bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03703e96a41b434ab9cd6c41f16914bb, State: Initialized, Role: FOLLOWER
I20260620 02:52:59.601480 8504 consensus_queue.cc:260] T ac28b4c3b78445efba3d63245d3f0fe9 P 03703e96a41b434ab9cd6c41f16914bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } }
I20260620 02:52:59.606107 8504 ts_tablet_manager.cc:1428] T ac28b4c3b78445efba3d63245d3f0fe9 P 03703e96a41b434ab9cd6c41f16914bb: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260620 02:52:59.608338 8500 tablet_bootstrap.cc:492] T ac28b4c3b78445efba3d63245d3f0fe9 P 6b60b7bcf07f449a98435b338b4a952c: No bootstrap required, opened a new log
I20260620 02:52:59.608489 8500 ts_tablet_manager.cc:1397] T ac28b4c3b78445efba3d63245d3f0fe9 P 6b60b7bcf07f449a98435b338b4a952c: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260620 02:52:59.609303 8500 raft_consensus.cc:348] T ac28b4c3b78445efba3d63245d3f0fe9 P 6b60b7bcf07f449a98435b338b4a952c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } }
I20260620 02:52:59.609566 8500 raft_consensus.cc:374] T ac28b4c3b78445efba3d63245d3f0fe9 P 6b60b7bcf07f449a98435b338b4a952c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:59.609656 8500 raft_consensus.cc:729] T ac28b4c3b78445efba3d63245d3f0fe9 P 6b60b7bcf07f449a98435b338b4a952c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b60b7bcf07f449a98435b338b4a952c, State: Initialized, Role: FOLLOWER
I20260620 02:52:59.609925 8500 consensus_queue.cc:260] T ac28b4c3b78445efba3d63245d3f0fe9 P 6b60b7bcf07f449a98435b338b4a952c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } }
I20260620 02:52:59.610623 8500 ts_tablet_manager.cc:1428] T ac28b4c3b78445efba3d63245d3f0fe9 P 6b60b7bcf07f449a98435b338b4a952c: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260620 02:52:59.611099 8501 tablet_bootstrap.cc:492] T ac28b4c3b78445efba3d63245d3f0fe9 P 8eeb98eed5c94cc898e55490f1ebd0cb: No bootstrap required, opened a new log
I20260620 02:52:59.611254 8501 ts_tablet_manager.cc:1397] T ac28b4c3b78445efba3d63245d3f0fe9 P 8eeb98eed5c94cc898e55490f1ebd0cb: Time spent bootstrapping tablet: real 0.016s user 0.015s sys 0.000s
I20260620 02:52:59.612175 8501 raft_consensus.cc:348] T ac28b4c3b78445efba3d63245d3f0fe9 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } }
I20260620 02:52:59.612406 8501 raft_consensus.cc:374] T ac28b4c3b78445efba3d63245d3f0fe9 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:59.612497 8501 raft_consensus.cc:729] T ac28b4c3b78445efba3d63245d3f0fe9 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eeb98eed5c94cc898e55490f1ebd0cb, State: Initialized, Role: FOLLOWER
I20260620 02:52:59.612740 8501 consensus_queue.cc:260] T ac28b4c3b78445efba3d63245d3f0fe9 P 8eeb98eed5c94cc898e55490f1ebd0cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } }
I20260620 02:52:59.613371 8501 ts_tablet_manager.cc:1428] T ac28b4c3b78445efba3d63245d3f0fe9 P 8eeb98eed5c94cc898e55490f1ebd0cb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 02:52:59.633332 8506 raft_consensus.cc:482] T ac28b4c3b78445efba3d63245d3f0fe9 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:52:59.633589 8506 raft_consensus.cc:504] T ac28b4c3b78445efba3d63245d3f0fe9 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } }
I20260620 02:52:59.634207 8506 leader_election.cc:288] T ac28b4c3b78445efba3d63245d3f0fe9 P 8eeb98eed5c94cc898e55490f1ebd0cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b60b7bcf07f449a98435b338b4a952c (127.0.97.1:36377), 03703e96a41b434ab9cd6c41f16914bb (127.0.97.3:35929)
I20260620 02:52:59.634720 8149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac28b4c3b78445efba3d63245d3f0fe9" candidate_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b60b7bcf07f449a98435b338b4a952c" is_pre_election: true
I20260620 02:52:59.634820 8426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac28b4c3b78445efba3d63245d3f0fe9" candidate_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03703e96a41b434ab9cd6c41f16914bb" is_pre_election: true
I20260620 02:52:59.634981 8149 raft_consensus.cc:2454] T ac28b4c3b78445efba3d63245d3f0fe9 P 6b60b7bcf07f449a98435b338b4a952c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eeb98eed5c94cc898e55490f1ebd0cb in term 0.
I20260620 02:52:59.635097 8426 raft_consensus.cc:2454] T ac28b4c3b78445efba3d63245d3f0fe9 P 03703e96a41b434ab9cd6c41f16914bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eeb98eed5c94cc898e55490f1ebd0cb in term 0.
I20260620 02:52:59.635529 8220 leader_election.cc:302] T ac28b4c3b78445efba3d63245d3f0fe9 P 8eeb98eed5c94cc898e55490f1ebd0cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03703e96a41b434ab9cd6c41f16914bb, 8eeb98eed5c94cc898e55490f1ebd0cb; no voters:
I20260620 02:52:59.635881 8506 raft_consensus.cc:2781] T ac28b4c3b78445efba3d63245d3f0fe9 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:52:59.636045 8506 raft_consensus.cc:482] T ac28b4c3b78445efba3d63245d3f0fe9 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:52:59.636144 8506 raft_consensus.cc:3037] T ac28b4c3b78445efba3d63245d3f0fe9 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:59.638823 8506 raft_consensus.cc:504] T ac28b4c3b78445efba3d63245d3f0fe9 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } }
I20260620 02:52:59.639351 8506 leader_election.cc:288] T ac28b4c3b78445efba3d63245d3f0fe9 P 8eeb98eed5c94cc898e55490f1ebd0cb [CANDIDATE]: Term 1 election: Requested vote from peers 6b60b7bcf07f449a98435b338b4a952c (127.0.97.1:36377), 03703e96a41b434ab9cd6c41f16914bb (127.0.97.3:35929)
I20260620 02:52:59.639809 8149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac28b4c3b78445efba3d63245d3f0fe9" candidate_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b60b7bcf07f449a98435b338b4a952c"
I20260620 02:52:59.640089 8149 raft_consensus.cc:3037] T ac28b4c3b78445efba3d63245d3f0fe9 P 6b60b7bcf07f449a98435b338b4a952c [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:59.640059 8426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac28b4c3b78445efba3d63245d3f0fe9" candidate_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03703e96a41b434ab9cd6c41f16914bb"
I20260620 02:52:59.640295 8426 raft_consensus.cc:3037] T ac28b4c3b78445efba3d63245d3f0fe9 P 03703e96a41b434ab9cd6c41f16914bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:59.643685 8149 raft_consensus.cc:2454] T ac28b4c3b78445efba3d63245d3f0fe9 P 6b60b7bcf07f449a98435b338b4a952c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eeb98eed5c94cc898e55490f1ebd0cb in term 1.
I20260620 02:52:59.643712 8426 raft_consensus.cc:2454] T ac28b4c3b78445efba3d63245d3f0fe9 P 03703e96a41b434ab9cd6c41f16914bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eeb98eed5c94cc898e55490f1ebd0cb in term 1.
I20260620 02:52:59.644177 8220 leader_election.cc:302] T ac28b4c3b78445efba3d63245d3f0fe9 P 8eeb98eed5c94cc898e55490f1ebd0cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03703e96a41b434ab9cd6c41f16914bb, 8eeb98eed5c94cc898e55490f1ebd0cb; no voters:
I20260620 02:52:59.644470 8506 raft_consensus.cc:2781] T ac28b4c3b78445efba3d63245d3f0fe9 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:59.645010 8506 raft_consensus.cc:686] T ac28b4c3b78445efba3d63245d3f0fe9 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 1 LEADER]: Becoming Leader. State: Replica: 8eeb98eed5c94cc898e55490f1ebd0cb, State: Running, Role: LEADER
I20260620 02:52:59.645666 8506 consensus_queue.cc:237] T ac28b4c3b78445efba3d63245d3f0fe9 P 8eeb98eed5c94cc898e55490f1ebd0cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } }
I20260620 02:52:59.649453 7988 catalog_manager.cc:5495] T ac28b4c3b78445efba3d63245d3f0fe9 P 8eeb98eed5c94cc898e55490f1ebd0cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 8eeb98eed5c94cc898e55490f1ebd0cb (127.0.97.2). New cstate: current_term: 1 leader_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } health_report { overall_health: UNKNOWN } } }
I20260620 02:52:59.677824 8546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:59.671295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42762 (local address 127.0.97.2:40485)
0620 02:52:59.671504 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:59.671511 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:59.671605 (+ 94us) server_negotiation.cc:407] Connection header received
0620 02:52:59.671822 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:59.671829 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:59.671937 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:59.672121 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:52:59.672467 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.673616 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.675388 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.676030 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.676159 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:59.676529 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:52:59.677234 (+ 705us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:52:59.677385 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:59.677442 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 02:52:59.677537 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:52:59.681452 8149 raft_consensus.cc:1261] T ac28b4c3b78445efba3d63245d3f0fe9 P 6b60b7bcf07f449a98435b338b4a952c [term 1 FOLLOWER]: Refusing update from remote peer 8eeb98eed5c94cc898e55490f1ebd0cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:59.681447 8426 raft_consensus.cc:1261] T ac28b4c3b78445efba3d63245d3f0fe9 P 03703e96a41b434ab9cd6c41f16914bb [term 1 FOLLOWER]: Refusing update from remote peer 8eeb98eed5c94cc898e55490f1ebd0cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:59.682123 8506 consensus_queue.cc:1035] T ac28b4c3b78445efba3d63245d3f0fe9 P 8eeb98eed5c94cc898e55490f1ebd0cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:52:59.682500 8591 consensus_queue.cc:1035] T ac28b4c3b78445efba3d63245d3f0fe9 P 8eeb98eed5c94cc898e55490f1ebd0cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:52:59.700290 7988 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:52:59.700528 7988 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:52:59.706332 8267 tablet_service.cc:1514] Processing DeleteTablet for tablet ac28b4c3b78445efba3d63245d3f0fe9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:59 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42746
I20260620 02:52:59.706529 8405 tablet_service.cc:1514] Processing DeleteTablet for tablet ac28b4c3b78445efba3d63245d3f0fe9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:59 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38360
I20260620 02:52:59.706861 8575 tablet_replica.cc:330] T ac28b4c3b78445efba3d63245d3f0fe9 P 8eeb98eed5c94cc898e55490f1ebd0cb: stopping tablet replica
I20260620 02:52:59.707086 7988 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4e7c66c5c1845aebeeedfe4dc6532f0: Sending DeleteTablet for 3 replicas of tablet ac28b4c3b78445efba3d63245d3f0fe9
I20260620 02:52:59.707063 8575 raft_consensus.cc:2229] T ac28b4c3b78445efba3d63245d3f0fe9 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:52:59.707595 8575 raft_consensus.cc:2258] T ac28b4c3b78445efba3d63245d3f0fe9 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:59.707854 8574 tablet_replica.cc:330] T ac28b4c3b78445efba3d63245d3f0fe9 P 03703e96a41b434ab9cd6c41f16914bb: stopping tablet replica
I20260620 02:52:59.707854 8267 tablet_service.cc:1514] Processing DeleteTablet for tablet ac28b4c3b78445efba3d63245d3f0fe9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:59 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42746
I20260620 02:52:59.708048 8574 raft_consensus.cc:2229] T ac28b4c3b78445efba3d63245d3f0fe9 P 03703e96a41b434ab9cd6c41f16914bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:52:59.708257 8574 raft_consensus.cc:2258] T ac28b4c3b78445efba3d63245d3f0fe9 P 03703e96a41b434ab9cd6c41f16914bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:59.709403 8405 tablet_service.cc:1514] Processing DeleteTablet for tablet ac28b4c3b78445efba3d63245d3f0fe9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:59 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38360
I20260620 02:52:59.709693 7990 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:52:59.710453 7990 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:52:59.710759 8575 ts_tablet_manager.cc:1905] T ac28b4c3b78445efba3d63245d3f0fe9 P 8eeb98eed5c94cc898e55490f1ebd0cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:59.712708 8574 ts_tablet_manager.cc:1905] T ac28b4c3b78445efba3d63245d3f0fe9 P 03703e96a41b434ab9cd6c41f16914bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:59.721966 8574 ts_tablet_manager.cc:1918] T ac28b4c3b78445efba3d63245d3f0fe9 P 03703e96a41b434ab9cd6c41f16914bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:52:59.722007 8126 tablet_service.cc:1467] Processing CreateTablet for tablet b8803452868f4abcade98a48032c2895 (DEFAULT_TABLE table=test-table [id=f98d6dad19444ff69b8f1a7d33da5598]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:59.722162 8574 log.cc:1196] T ac28b4c3b78445efba3d63245d3f0fe9 P 03703e96a41b434ab9cd6c41f16914bb: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-2/wal/wals/ac28b4c3b78445efba3d63245d3f0fe9
I20260620 02:52:59.722589 8126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8803452868f4abcade98a48032c2895. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:59.722709 8574 ts_tablet_manager.cc:1939] T ac28b4c3b78445efba3d63245d3f0fe9 P 03703e96a41b434ab9cd6c41f16914bb: Deleting consensus metadata
W20260620 02:52:59.727453 7977 catalog_manager.cc:4808] TS 03703e96a41b434ab9cd6c41f16914bb (127.0.97.3:35929): delete failed for tablet ac28b4c3b78445efba3d63245d3f0fe9 because the tablet was not found. No further retry: Not found: Tablet not found: ac28b4c3b78445efba3d63245d3f0fe9
I20260620 02:52:59.727643 8267 tablet_service.cc:1467] Processing CreateTablet for tablet b8803452868f4abcade98a48032c2895 (DEFAULT_TABLE table=test-table [id=f98d6dad19444ff69b8f1a7d33da5598]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:59.727979 8405 tablet_service.cc:1467] Processing CreateTablet for tablet b8803452868f4abcade98a48032c2895 (DEFAULT_TABLE table=test-table [id=f98d6dad19444ff69b8f1a7d33da5598]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:52:59.728098 8267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8803452868f4abcade98a48032c2895. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:59.728379 8405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8803452868f4abcade98a48032c2895. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:52:59.730589 8127 tablet_service.cc:1514] Processing DeleteTablet for tablet ac28b4c3b78445efba3d63245d3f0fe9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:59 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36408
I20260620 02:52:59.733232 8573 tablet_replica.cc:330] T ac28b4c3b78445efba3d63245d3f0fe9 P 6b60b7bcf07f449a98435b338b4a952c: stopping tablet replica
I20260620 02:52:59.733439 8573 raft_consensus.cc:2229] T ac28b4c3b78445efba3d63245d3f0fe9 P 6b60b7bcf07f449a98435b338b4a952c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:52:59.733670 8573 raft_consensus.cc:2258] T ac28b4c3b78445efba3d63245d3f0fe9 P 6b60b7bcf07f449a98435b338b4a952c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:59.733978 8500 tablet_bootstrap.cc:492] T b8803452868f4abcade98a48032c2895 P 6b60b7bcf07f449a98435b338b4a952c: Bootstrap starting.
I20260620 02:52:59.735337 8500 tablet_bootstrap.cc:654] T b8803452868f4abcade98a48032c2895 P 6b60b7bcf07f449a98435b338b4a952c: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:59.735685 8501 tablet_bootstrap.cc:492] T b8803452868f4abcade98a48032c2895 P 8eeb98eed5c94cc898e55490f1ebd0cb: Bootstrap starting.
I20260620 02:52:59.737489 8573 ts_tablet_manager.cc:1905] T ac28b4c3b78445efba3d63245d3f0fe9 P 6b60b7bcf07f449a98435b338b4a952c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:59.737936 8501 tablet_bootstrap.cc:654] T b8803452868f4abcade98a48032c2895 P 8eeb98eed5c94cc898e55490f1ebd0cb: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:59.764290 8575 ts_tablet_manager.cc:1918] T ac28b4c3b78445efba3d63245d3f0fe9 P 8eeb98eed5c94cc898e55490f1ebd0cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:52:59.764503 8575 log.cc:1196] T ac28b4c3b78445efba3d63245d3f0fe9 P 8eeb98eed5c94cc898e55490f1ebd0cb: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-1/wal/wals/ac28b4c3b78445efba3d63245d3f0fe9
I20260620 02:52:59.765070 8575 ts_tablet_manager.cc:1939] T ac28b4c3b78445efba3d63245d3f0fe9 P 8eeb98eed5c94cc898e55490f1ebd0cb: Deleting consensus metadata
I20260620 02:52:59.770316 8504 tablet_bootstrap.cc:492] T b8803452868f4abcade98a48032c2895 P 03703e96a41b434ab9cd6c41f16914bb: Bootstrap starting.
I20260620 02:52:59.770926 8573 ts_tablet_manager.cc:1918] T ac28b4c3b78445efba3d63245d3f0fe9 P 6b60b7bcf07f449a98435b338b4a952c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:52:59.771087 8573 log.cc:1196] T ac28b4c3b78445efba3d63245d3f0fe9 P 6b60b7bcf07f449a98435b338b4a952c: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-0/wal/wals/ac28b4c3b78445efba3d63245d3f0fe9
I20260620 02:52:59.771515 8573 ts_tablet_manager.cc:1939] T ac28b4c3b78445efba3d63245d3f0fe9 P 6b60b7bcf07f449a98435b338b4a952c: Deleting consensus metadata
I20260620 02:52:59.772294 8504 tablet_bootstrap.cc:654] T b8803452868f4abcade98a48032c2895 P 03703e96a41b434ab9cd6c41f16914bb: Neither blocks nor log segments found. Creating new log.
I20260620 02:52:59.774537 7977 catalog_manager.cc:4841] TS 6b60b7bcf07f449a98435b338b4a952c (127.0.97.1:36377): tablet ac28b4c3b78445efba3d63245d3f0fe9 (table test-table [id=4029f623d9a94599ba240725c7d12fd3]) successfully deleted
I20260620 02:52:59.777149 8500 tablet_bootstrap.cc:492] T b8803452868f4abcade98a48032c2895 P 6b60b7bcf07f449a98435b338b4a952c: No bootstrap required, opened a new log
I20260620 02:52:59.777328 8500 ts_tablet_manager.cc:1397] T b8803452868f4abcade98a48032c2895 P 6b60b7bcf07f449a98435b338b4a952c: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.005s
I20260620 02:52:59.779435 8500 raft_consensus.cc:348] T b8803452868f4abcade98a48032c2895 P 6b60b7bcf07f449a98435b338b4a952c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } }
I20260620 02:52:59.779762 8500 raft_consensus.cc:374] T b8803452868f4abcade98a48032c2895 P 6b60b7bcf07f449a98435b338b4a952c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:59.779844 8500 raft_consensus.cc:729] T b8803452868f4abcade98a48032c2895 P 6b60b7bcf07f449a98435b338b4a952c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b60b7bcf07f449a98435b338b4a952c, State: Initialized, Role: FOLLOWER
I20260620 02:52:59.780143 8500 consensus_queue.cc:260] T b8803452868f4abcade98a48032c2895 P 6b60b7bcf07f449a98435b338b4a952c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } }
I20260620 02:52:59.780793 8500 ts_tablet_manager.cc:1428] T b8803452868f4abcade98a48032c2895 P 6b60b7bcf07f449a98435b338b4a952c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20260620 02:52:59.781570 7976 catalog_manager.cc:4808] TS 8eeb98eed5c94cc898e55490f1ebd0cb (127.0.97.2:40485): delete failed for tablet ac28b4c3b78445efba3d63245d3f0fe9 because the tablet was not found. No further retry: Not found: Tablet not found: ac28b4c3b78445efba3d63245d3f0fe9
I20260620 02:52:59.783164 8501 tablet_bootstrap.cc:492] T b8803452868f4abcade98a48032c2895 P 8eeb98eed5c94cc898e55490f1ebd0cb: No bootstrap required, opened a new log
I20260620 02:52:59.783356 8501 ts_tablet_manager.cc:1397] T b8803452868f4abcade98a48032c2895 P 8eeb98eed5c94cc898e55490f1ebd0cb: Time spent bootstrapping tablet: real 0.048s user 0.015s sys 0.020s
I20260620 02:52:59.784163 8501 raft_consensus.cc:348] T b8803452868f4abcade98a48032c2895 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } }
I20260620 02:52:59.784435 8501 raft_consensus.cc:374] T b8803452868f4abcade98a48032c2895 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:59.786980 8501 raft_consensus.cc:729] T b8803452868f4abcade98a48032c2895 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eeb98eed5c94cc898e55490f1ebd0cb, State: Initialized, Role: FOLLOWER
I20260620 02:52:59.787328 8501 consensus_queue.cc:260] T b8803452868f4abcade98a48032c2895 P 8eeb98eed5c94cc898e55490f1ebd0cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } }
I20260620 02:52:59.787885 8501 ts_tablet_manager.cc:1428] T b8803452868f4abcade98a48032c2895 P 8eeb98eed5c94cc898e55490f1ebd0cb: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260620 02:52:59.788790 8504 tablet_bootstrap.cc:492] T b8803452868f4abcade98a48032c2895 P 03703e96a41b434ab9cd6c41f16914bb: No bootstrap required, opened a new log
I20260620 02:52:59.789007 8504 ts_tablet_manager.cc:1397] T b8803452868f4abcade98a48032c2895 P 03703e96a41b434ab9cd6c41f16914bb: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.011s
I20260620 02:52:59.789943 8504 raft_consensus.cc:348] T b8803452868f4abcade98a48032c2895 P 03703e96a41b434ab9cd6c41f16914bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } }
I20260620 02:52:59.790266 8504 raft_consensus.cc:374] T b8803452868f4abcade98a48032c2895 P 03703e96a41b434ab9cd6c41f16914bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:52:59.790429 8504 raft_consensus.cc:729] T b8803452868f4abcade98a48032c2895 P 03703e96a41b434ab9cd6c41f16914bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03703e96a41b434ab9cd6c41f16914bb, State: Initialized, Role: FOLLOWER
I20260620 02:52:59.790719 8504 consensus_queue.cc:260] T b8803452868f4abcade98a48032c2895 P 03703e96a41b434ab9cd6c41f16914bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } }
I20260620 02:52:59.791252 8504 ts_tablet_manager.cc:1428] T b8803452868f4abcade98a48032c2895 P 03703e96a41b434ab9cd6c41f16914bb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 02:52:59.855855 8506 raft_consensus.cc:482] T b8803452868f4abcade98a48032c2895 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:52:59.856078 8506 raft_consensus.cc:504] T b8803452868f4abcade98a48032c2895 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } }
I20260620 02:52:59.856788 8506 leader_election.cc:288] T b8803452868f4abcade98a48032c2895 P 8eeb98eed5c94cc898e55490f1ebd0cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b60b7bcf07f449a98435b338b4a952c (127.0.97.1:36377), 03703e96a41b434ab9cd6c41f16914bb (127.0.97.3:35929)
I20260620 02:52:59.857342 8426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8803452868f4abcade98a48032c2895" candidate_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03703e96a41b434ab9cd6c41f16914bb" is_pre_election: true
I20260620 02:52:59.857640 8426 raft_consensus.cc:2454] T b8803452868f4abcade98a48032c2895 P 03703e96a41b434ab9cd6c41f16914bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eeb98eed5c94cc898e55490f1ebd0cb in term 0.
I20260620 02:52:59.858013 8149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8803452868f4abcade98a48032c2895" candidate_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b60b7bcf07f449a98435b338b4a952c" is_pre_election: true
I20260620 02:52:59.858129 8220 leader_election.cc:302] T b8803452868f4abcade98a48032c2895 P 8eeb98eed5c94cc898e55490f1ebd0cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03703e96a41b434ab9cd6c41f16914bb, 8eeb98eed5c94cc898e55490f1ebd0cb; no voters:
I20260620 02:52:59.858276 8149 raft_consensus.cc:2454] T b8803452868f4abcade98a48032c2895 P 6b60b7bcf07f449a98435b338b4a952c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eeb98eed5c94cc898e55490f1ebd0cb in term 0.
I20260620 02:52:59.858597 8506 raft_consensus.cc:2781] T b8803452868f4abcade98a48032c2895 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:52:59.858724 8506 raft_consensus.cc:482] T b8803452868f4abcade98a48032c2895 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:52:59.858820 8506 raft_consensus.cc:3037] T b8803452868f4abcade98a48032c2895 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:59.861287 8506 raft_consensus.cc:504] T b8803452868f4abcade98a48032c2895 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } }
I20260620 02:52:59.861842 8506 leader_election.cc:288] T b8803452868f4abcade98a48032c2895 P 8eeb98eed5c94cc898e55490f1ebd0cb [CANDIDATE]: Term 1 election: Requested vote from peers 6b60b7bcf07f449a98435b338b4a952c (127.0.97.1:36377), 03703e96a41b434ab9cd6c41f16914bb (127.0.97.3:35929)
I20260620 02:52:59.862345 8149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8803452868f4abcade98a48032c2895" candidate_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b60b7bcf07f449a98435b338b4a952c"
I20260620 02:52:59.862627 8149 raft_consensus.cc:3037] T b8803452868f4abcade98a48032c2895 P 6b60b7bcf07f449a98435b338b4a952c [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:59.864530 8149 raft_consensus.cc:2454] T b8803452868f4abcade98a48032c2895 P 6b60b7bcf07f449a98435b338b4a952c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eeb98eed5c94cc898e55490f1ebd0cb in term 1.
I20260620 02:52:59.864877 8426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8803452868f4abcade98a48032c2895" candidate_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03703e96a41b434ab9cd6c41f16914bb"
I20260620 02:52:59.865007 8220 leader_election.cc:302] T b8803452868f4abcade98a48032c2895 P 8eeb98eed5c94cc898e55490f1ebd0cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b60b7bcf07f449a98435b338b4a952c, 8eeb98eed5c94cc898e55490f1ebd0cb; no voters:
I20260620 02:52:59.865125 8426 raft_consensus.cc:3037] T b8803452868f4abcade98a48032c2895 P 03703e96a41b434ab9cd6c41f16914bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:52:59.865303 8506 raft_consensus.cc:2781] T b8803452868f4abcade98a48032c2895 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:52:59.865789 8506 raft_consensus.cc:686] T b8803452868f4abcade98a48032c2895 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 1 LEADER]: Becoming Leader. State: Replica: 8eeb98eed5c94cc898e55490f1ebd0cb, State: Running, Role: LEADER
I20260620 02:52:59.866074 8506 consensus_queue.cc:237] T b8803452868f4abcade98a48032c2895 P 8eeb98eed5c94cc898e55490f1ebd0cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } }
I20260620 02:52:59.868523 8426 raft_consensus.cc:2454] T b8803452868f4abcade98a48032c2895 P 03703e96a41b434ab9cd6c41f16914bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eeb98eed5c94cc898e55490f1ebd0cb in term 1.
I20260620 02:52:59.871158 7988 catalog_manager.cc:5495] T b8803452868f4abcade98a48032c2895 P 8eeb98eed5c94cc898e55490f1ebd0cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 8eeb98eed5c94cc898e55490f1ebd0cb (127.0.97.2). New cstate: current_term: 1 leader_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8eeb98eed5c94cc898e55490f1ebd0cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 } health_report { overall_health: UNKNOWN } } }
I20260620 02:52:59.920408 8604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:59.901617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47457 (local address 127.0.97.3:35929)
0620 02:52:59.911230 (+ 9613us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:59.911235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:59.914904 (+ 3669us) server_negotiation.cc:407] Connection header received
0620 02:52:59.915189 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:59.915193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:59.915260 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:59.915369 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:52:59.915918 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.916657 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.918479 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.919064 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.919189 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:59.919639 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:52:59.919738 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:52:59.919830 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:59.919995 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 02:52:59.920067 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9498,"thread_start_us":9364,"threads_started":1}
I20260620 02:52:59.920526 8603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:59.901520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35929 (local address 127.0.97.2:47457)
0620 02:52:59.914752 (+ 13232us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:59.914771 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:59.915001 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:59.915389 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:59.915394 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:59.915405 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:52:59.915759 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:59.915768 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.916796 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.916802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:59.918283 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:59.918295 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.919242 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.919251 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:59.919370 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:59.919433 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:52:59.919845 (+ 412us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:52:59.919862 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 02:52:59.919977 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 02:52:59.920108 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13064,"thread_start_us":13415,"threads_started":1}
I20260620 02:52:59.929003 8426 raft_consensus.cc:1261] T b8803452868f4abcade98a48032c2895 P 03703e96a41b434ab9cd6c41f16914bb [term 1 FOLLOWER]: Refusing update from remote peer 8eeb98eed5c94cc898e55490f1ebd0cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:59.929051 8149 raft_consensus.cc:1261] T b8803452868f4abcade98a48032c2895 P 6b60b7bcf07f449a98435b338b4a952c [term 1 FOLLOWER]: Refusing update from remote peer 8eeb98eed5c94cc898e55490f1ebd0cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:52:59.929615 8591 consensus_queue.cc:1035] T b8803452868f4abcade98a48032c2895 P 8eeb98eed5c94cc898e55490f1ebd0cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b60b7bcf07f449a98435b338b4a952c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:52:59.929886 8506 consensus_queue.cc:1035] T b8803452868f4abcade98a48032c2895 P 8eeb98eed5c94cc898e55490f1ebd0cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "03703e96a41b434ab9cd6c41f16914bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:52:59.962646 8546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:59.955850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41151 (local address 127.0.97.2:40485)
0620 02:52:59.956055 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 02:52:59.956063 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:52:59.956085 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:52:59.956167 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:52:59.956176 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:52:59.956284 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:52:59.956406 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:52:59.956870 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.958078 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.960440 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.961120 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.961215 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:59.961770 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:52:59.961913 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:52:59.961972 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 02:52:59.962186 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 02:52:59.962270 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:52:59.963550 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:52:59.955305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40485 (local address 127.0.97.3:41151)
0620 02:52:59.955838 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 02:52:59.955866 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 02:52:59.955981 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:52:59.956432 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:52:59.956438 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:52:59.956451 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 02:52:59.956705 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:59.956717 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.958258 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.958263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:59.960282 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:52:59.960297 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:52:59.961374 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:52:59.961380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:52:59.961560 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:52:59.961605 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 02:52:59.962051 (+ 446us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 02:52:59.962060 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 02:52:59.963073 (+ 1013us) client_negotiation.cc:240] Negotiation successful
0620 02:52:59.963224 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"threads_started":1}
I20260620 02:52:59.991887 7988 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:52:59.992138 7988 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:52:59.994933 7988 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4e7c66c5c1845aebeeedfe4dc6532f0: Sending DeleteTablet for 3 replicas of tablet b8803452868f4abcade98a48032c2895
I20260620 02:52:59.995761 8127 tablet_service.cc:1514] Processing DeleteTablet for tablet b8803452868f4abcade98a48032c2895 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:59 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36408
I20260620 02:52:59.995824 8267 tablet_service.cc:1514] Processing DeleteTablet for tablet b8803452868f4abcade98a48032c2895 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:59 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42746
I20260620 02:52:59.995944 8405 tablet_service.cc:1514] Processing DeleteTablet for tablet b8803452868f4abcade98a48032c2895 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:52:59 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38360
I20260620 02:52:59.996158 8573 tablet_replica.cc:330] T b8803452868f4abcade98a48032c2895 P 6b60b7bcf07f449a98435b338b4a952c: stopping tablet replica
I20260620 02:52:59.996155 8575 tablet_replica.cc:330] T b8803452868f4abcade98a48032c2895 P 8eeb98eed5c94cc898e55490f1ebd0cb: stopping tablet replica
I20260620 02:52:59.996227 8574 tablet_replica.cc:330] T b8803452868f4abcade98a48032c2895 P 03703e96a41b434ab9cd6c41f16914bb: stopping tablet replica
I20260620 02:52:59.996322 8573 raft_consensus.cc:2229] T b8803452868f4abcade98a48032c2895 P 6b60b7bcf07f449a98435b338b4a952c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:52:59.996361 8575 raft_consensus.cc:2229] T b8803452868f4abcade98a48032c2895 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:52:59.996387 8574 raft_consensus.cc:2229] T b8803452868f4abcade98a48032c2895 P 03703e96a41b434ab9cd6c41f16914bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:52:59.996529 8573 raft_consensus.cc:2258] T b8803452868f4abcade98a48032c2895 P 6b60b7bcf07f449a98435b338b4a952c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:59.996577 8574 raft_consensus.cc:2258] T b8803452868f4abcade98a48032c2895 P 03703e96a41b434ab9cd6c41f16914bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:59.996685 8575 raft_consensus.cc:2258] T b8803452868f4abcade98a48032c2895 P 8eeb98eed5c94cc898e55490f1ebd0cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:52:59.997771 8574 ts_tablet_manager.cc:1905] T b8803452868f4abcade98a48032c2895 P 03703e96a41b434ab9cd6c41f16914bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:52:59.997879 8575 ts_tablet_manager.cc:1905] T b8803452868f4abcade98a48032c2895 P 8eeb98eed5c94cc898e55490f1ebd0cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:53:00.002592 8573 ts_tablet_manager.cc:1905] T b8803452868f4abcade98a48032c2895 P 6b60b7bcf07f449a98435b338b4a952c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:53:00.004037 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 8061
I20260620 02:53:00.005790 8574 ts_tablet_manager.cc:1918] T b8803452868f4abcade98a48032c2895 P 03703e96a41b434ab9cd6c41f16914bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:53:00.005966 8574 log.cc:1196] T b8803452868f4abcade98a48032c2895 P 03703e96a41b434ab9cd6c41f16914bb: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-2/wal/wals/b8803452868f4abcade98a48032c2895
I20260620 02:53:00.006413 8575 ts_tablet_manager.cc:1918] T b8803452868f4abcade98a48032c2895 P 8eeb98eed5c94cc898e55490f1ebd0cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:53:00.006466 8574 ts_tablet_manager.cc:1939] T b8803452868f4abcade98a48032c2895 P 03703e96a41b434ab9cd6c41f16914bb: Deleting consensus metadata
I20260620 02:53:00.006628 8575 log.cc:1196] T b8803452868f4abcade98a48032c2895 P 8eeb98eed5c94cc898e55490f1ebd0cb: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-1/wal/wals/b8803452868f4abcade98a48032c2895
I20260620 02:53:00.007077 8575 ts_tablet_manager.cc:1939] T b8803452868f4abcade98a48032c2895 P 8eeb98eed5c94cc898e55490f1ebd0cb: Deleting consensus metadata
I20260620 02:53:00.007692 7977 catalog_manager.cc:4841] TS 03703e96a41b434ab9cd6c41f16914bb (127.0.97.3:35929): tablet b8803452868f4abcade98a48032c2895 (table test-table [id=f98d6dad19444ff69b8f1a7d33da5598]) successfully deleted
I20260620 02:53:00.010236 7976 catalog_manager.cc:4841] TS 8eeb98eed5c94cc898e55490f1ebd0cb (127.0.97.2:40485): tablet b8803452868f4abcade98a48032c2895 (table test-table [id=f98d6dad19444ff69b8f1a7d33da5598]) successfully deleted
I20260620 02:53:00.013008 8573 ts_tablet_manager.cc:1918] T b8803452868f4abcade98a48032c2895 P 6b60b7bcf07f449a98435b338b4a952c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 02:53:00.013142 8573 log.cc:1196] T b8803452868f4abcade98a48032c2895 P 6b60b7bcf07f449a98435b338b4a952c: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781923914418765-388-0/minicluster-data/ts-0/wal/wals/b8803452868f4abcade98a48032c2895
I20260620 02:53:00.013514 8573 ts_tablet_manager.cc:1939] T b8803452868f4abcade98a48032c2895 P 6b60b7bcf07f449a98435b338b4a952c: Deleting consensus metadata
I20260620 02:53:00.014492 7977 catalog_manager.cc:4841] TS 6b60b7bcf07f449a98435b338b4a952c (127.0.97.1:36377): tablet b8803452868f4abcade98a48032c2895 (table test-table [id=f98d6dad19444ff69b8f1a7d33da5598]) successfully deleted
Jun 20 02:53:00 dist-test-slave-8p8c krb5kdc[7945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260620 02:53:00.029639 8612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:00.005503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36422 (local address 127.0.97.1:36377)
0620 02:53:00.013975 (+ 8472us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:00.013979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:00.014113 (+ 134us) server_negotiation.cc:407] Connection header received
0620 02:53:00.014332 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:00.014355 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:00.014471 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:00.014577 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:00.015831 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:00.016548 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:00.018028 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:00.018628 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:00.018781 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:00.023524 (+ 4743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:00.023539 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:00.023552 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:53:00.023593 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:00.026561 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:00.027258 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:00.027267 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:00.027269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:00.027404 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:00.027777 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:00.027783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:00.027786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:00.028138 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:53:00.028681 (+ 543us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:00.029030 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 02:53:00.029254 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8359,"thread_start_us":8259,"threads_started":1}
I20260620 02:53:00.030316 8192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:53:00.156168 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 8061
I20260620 02:53:00.188177 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 8201
Jun 20 02:53:00 dist-test-slave-8p8c krb5kdc[7945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260620 02:53:00.206409 8546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:00.189046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42764 (local address 127.0.97.2:40485)
0620 02:53:00.189241 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:00.189252 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:00.189414 (+ 162us) server_negotiation.cc:407] Connection header received
0620 02:53:00.189643 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:00.189656 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:00.189733 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:00.189857 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:00.191416 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:00.192458 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:00.193896 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:00.194477 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:00.194623 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:00.199136 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:00.199150 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:00.199170 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:53:00.199215 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:00.203349 (+ 4134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:00.204100 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:00.204111 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:00.204114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:00.204263 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:00.204738 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:00.204742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:00.204746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:00.205217 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:53:00.205517 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:00.205778 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 02:53:00.206014 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:53:00.207199 8332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:53:00.365582 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 8201
I20260620 02:53:00.386139 8615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:00.385249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36377 (local address 127.0.97.3:51863)
0620 02:53:00.385777 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:00.385905 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":273,"threads_started":1}
W20260620 02:53:00.386516 8360 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36377: connect: Connection refused (error 111)
I20260620 02:53:00.387786 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 8341
I20260620 02:53:00.388391 8615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:00.387925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36377 (local address 127.0.97.3:34771)
0620 02:53:00.388153 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:00.388236 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260620 02:53:00.388665 8360 consensus_peers.cc:487] T 8a0a8453266f4edeb7db67a1db120b75 P 03703e96a41b434ab9cd6c41f16914bb -> Peer 6b60b7bcf07f449a98435b338b4a952c (127.0.97.1:36377): Couldn't send request to peer 6b60b7bcf07f449a98435b338b4a952c. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36377: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Jun 20 02:53:00 dist-test-slave-8p8c krb5kdc[7945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260620 02:53:00.403494 8604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:00.388696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38382 (local address 127.0.97.3:35929)
0620 02:53:00.388858 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:00.388863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:00.388884 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:53:00.389231 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:00.389235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:00.389293 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:00.389374 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:00.390724 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:00.391460 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:00.392744 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:00.393242 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:00.393340 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:00.397639 (+ 4299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:00.397650 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:00.397666 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:53:00.397702 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:00.400737 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:00.401391 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:00.401397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:00.401400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:00.401554 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:00.401975 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:00.401979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:00.401981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:00.402340 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:53:00.402678 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:00.402932 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 02:53:00.403224 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 02:53:00.404165 8471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:53:00.531674 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 8341
I20260620 02:53:00.553517 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 7958
I20260620 02:53:00.554714 8030 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:53:00.668023 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 7958
2026-06-20T02:53:00Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5922 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:53:00 dist-test-slave-8p8c krb5kdc[8621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:53:00 dist-test-slave-8p8c krb5kdc[8621](info): set up 2 sockets
Jun 20 02:53:00 dist-test-slave-8p8c krb5kdc[8621](info): commencing operation
krb5kdc: starting...
W20260620 02:53:01.791481 388 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.006s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:53:01 dist-test-slave-8p8c krb5kdc[8621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:53:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:53:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:53:02.026476 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41827
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:53:02.346658 8634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:53:02.346935 8634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:53:02.347100 8634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:53:02.354045 8634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:53:02.354148 8634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:53:02.354239 8634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:53:02.354342 8634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:53:02.354423 8634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:53:02.363867 8634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:53:02.365118 8634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:53:02.366454 8634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:53:02.378229 8640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:02.378428 8639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:02.378428 8642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:53:02.378746 8634 server_base.cc:1034] running on GCE node
I20260620 02:53:02.379575 8634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:53:02.380892 8634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:53:02.382115 8634 hybrid_clock.cc:648] HybridClock initialized: now 1781923982382080 us; error 33 us; skew 500 ppm
I20260620 02:53:02.382498 8634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T02:53:02Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1798 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:53:02 dist-test-slave-8p8c krb5kdc[8648](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:53:02 dist-test-slave-8p8c krb5kdc[8648](info): set up 2 sockets
Jun 20 02:53:02 dist-test-slave-8p8c krb5kdc[8648](info): commencing operation
krb5kdc: starting...
W20260620 02:53:03.635846 388 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:53:03 dist-test-slave-8p8c krb5kdc[8648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:53:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:53:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:53:03.863333 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45373
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:53:04.205274 8661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:53:04.205595 8661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:53:04.205761 8661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:53:04.212795 8661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:53:04.212941 8661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:53:04.213059 8661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:53:04.213153 8661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:53:04.213238 8661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:53:04.223292 8661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:53:04.224728 8661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:53:04.226046 8661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:53:04.238410 8666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:04.238372 8669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:04.238693 8667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:53:04.239079 8661 server_base.cc:1034] running on GCE node
I20260620 02:53:04.239822 8661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:53:04.241290 8661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:53:04.242496 8661 hybrid_clock.cc:648] HybridClock initialized: now 1781923984242423 us; error 42 us; skew 500 ppm
I20260620 02:53:04.242841 8661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:53:04 dist-test-slave-8p8c krb5kdc[8648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:53:04.250087 8661 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:53:04.251714 8661 webserver.cc:460] Webserver started at http://127.0.97.62:38185/ using document root <none> and password file <none>
I20260620 02:53:04.252410 8661 fs_manager.cc:359] Metadata directory not provided
I20260620 02:53:04.252565 8661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:53:04.252871 8661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:53:04.255885 8661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "b5002715c3ff480796ba9a8f7a239be6"
format_stamp: "Formatted at 2026-06-20 02:53:04 on dist-test-slave-8p8c"
server_key: "50617a24533db6e5b15fc35e8fab94c0"
server_key_iv: "169afd27b0b38a90653b75f1746322bf"
server_key_version: "encryptionkey@0"
I20260620 02:53:04.256661 8661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "b5002715c3ff480796ba9a8f7a239be6"
format_stamp: "Formatted at 2026-06-20 02:53:04 on dist-test-slave-8p8c"
server_key: "50617a24533db6e5b15fc35e8fab94c0"
server_key_iv: "169afd27b0b38a90653b75f1746322bf"
server_key_version: "encryptionkey@0"
I20260620 02:53:04.261667 8661 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 02:53:04.265120 8676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:04.266537 8661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 02:53:04.266728 8661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "b5002715c3ff480796ba9a8f7a239be6"
format_stamp: "Formatted at 2026-06-20 02:53:04 on dist-test-slave-8p8c"
server_key: "50617a24533db6e5b15fc35e8fab94c0"
server_key_iv: "169afd27b0b38a90653b75f1746322bf"
server_key_version: "encryptionkey@0"
I20260620 02:53:04.266911 8661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:53:04.308265 8661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:53:04.312739 8661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:53:04.313031 8661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:53:04.330493 8661 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45373
I20260620 02:53:04.330513 8727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45373 every 8 connection(s)
I20260620 02:53:04.332013 8661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:53:04.335707 8728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:53:04.336215 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 8661
I20260620 02:53:04.336412 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:53:04.337236 388 external_mini_cluster.cc:1428] Setting key 7a4b500e79179ccf9b75e974a581beea
I20260620 02:53:04.348717 8728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6: Bootstrap starting.
Jun 20 02:53:04 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:53:04.351785 8728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6: Neither blocks nor log segments found. Creating new log.
I20260620 02:53:04.352843 8728 log.cc:824] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6: Log is configured to *not* fsync() on all Append() calls
I20260620 02:53:04.355737 8728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6: No bootstrap required, opened a new log
I20260620 02:53:04.359685 8731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:04.339392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38804 (local address 127.0.97.62:45373)
0620 02:53:04.340145 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:04.340157 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:04.340201 (+ 44us) server_negotiation.cc:407] Connection header received
0620 02:53:04.342350 (+ 2149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:04.342452 (+ 102us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:04.342867 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:04.343294 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:04.344751 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:04.346120 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:04.347462 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:04.348139 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:04.348337 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:04.352405 (+ 4068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:04.352413 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:04.352428 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:53:04.352469 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:04.355582 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:04.356360 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:04.356366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:04.356373 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:04.356538 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:04.356920 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:04.356929 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:04.356931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:04.357469 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:53:04.357802 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:04.358206 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 02:53:04.358690 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":360,"threads_started":1}
I20260620 02:53:04.363612 8728 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5002715c3ff480796ba9a8f7a239be6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45373 } }
I20260620 02:53:04.363970 8728 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:53:04.364078 8728 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5002715c3ff480796ba9a8f7a239be6, State: Initialized, Role: FOLLOWER
I20260620 02:53:04.364593 8728 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5002715c3ff480796ba9a8f7a239be6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45373 } }
I20260620 02:53:04.364828 8728 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:53:04.364964 8728 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:53:04.365131 8728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:53:04.367591 8728 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5002715c3ff480796ba9a8f7a239be6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45373 } }
I20260620 02:53:04.368053 8728 leader_election.cc:302] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5002715c3ff480796ba9a8f7a239be6; no voters:
I20260620 02:53:04.368623 8728 leader_election.cc:288] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:53:04.368942 8733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:53:04.369585 8733 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [term 1 LEADER]: Becoming Leader. State: Replica: b5002715c3ff480796ba9a8f7a239be6, State: Running, Role: LEADER
I20260620 02:53:04.370155 8733 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5002715c3ff480796ba9a8f7a239be6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45373 } }
I20260620 02:53:04.370647 8728 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:53:04.373479 8735 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5002715c3ff480796ba9a8f7a239be6. Latest consensus state: current_term: 1 leader_uuid: "b5002715c3ff480796ba9a8f7a239be6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5002715c3ff480796ba9a8f7a239be6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45373 } } }
I20260620 02:53:04.373847 8735 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [sys.catalog]: This master's current role is: LEADER
I20260620 02:53:04.373462 8734 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5002715c3ff480796ba9a8f7a239be6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5002715c3ff480796ba9a8f7a239be6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45373 } } }
I20260620 02:53:04.374408 8734 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [sys.catalog]: This master's current role is: LEADER
I20260620 02:53:04.375430 8739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:53:04.379812 8739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:53:04.388217 8739 catalog_manager.cc:1347] Generated new cluster ID: 44d1789f4ba542fda3664099ea0223b1
I20260620 02:53:04.388327 8739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:53:04.397560 8739 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:53:04.399482 8739 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:53:04.407069 8739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6: Generated new TSK 0
I20260620 02:53:04.408089 8739 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:53:04.467639 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45373
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:53:04.796890 8754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:53:04.797209 8754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:53:04.797372 8754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:53:04.804711 8754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:53:04.804864 8754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:53:04.805037 8754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:53:04.814777 8754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45373
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:53:04.816056 8754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:53:04.817397 8754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:53:04.831086 8759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:04.831009 8762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:04.831046 8760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:53:04.831933 8754 server_base.cc:1034] running on GCE node
I20260620 02:53:04.832531 8754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:53:04.833488 8754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:53:04.834720 8754 hybrid_clock.cc:648] HybridClock initialized: now 1781923984834670 us; error 36 us; skew 500 ppm
I20260620 02:53:04.835078 8754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:53:04 dist-test-slave-8p8c krb5kdc[8648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:53:04.838179 8754 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:53:04.839726 8754 webserver.cc:460] Webserver started at http://127.0.97.1:43557/ using document root <none> and password file <none>
I20260620 02:53:04.840430 8754 fs_manager.cc:359] Metadata directory not provided
I20260620 02:53:04.840552 8754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:53:04.840808 8754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:53:04.843701 8754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-0/data/instance:
uuid: "6b9e385ed07f46599cb8c337f468777b"
format_stamp: "Formatted at 2026-06-20 02:53:04 on dist-test-slave-8p8c"
server_key: "275de7831ea17e55c725a19bbdefd97b"
server_key_iv: "1deed52e70f89634002cade5199fb5ae"
server_key_version: "encryptionkey@0"
I20260620 02:53:04.844441 8754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6b9e385ed07f46599cb8c337f468777b"
format_stamp: "Formatted at 2026-06-20 02:53:04 on dist-test-slave-8p8c"
server_key: "275de7831ea17e55c725a19bbdefd97b"
server_key_iv: "1deed52e70f89634002cade5199fb5ae"
server_key_version: "encryptionkey@0"
I20260620 02:53:04.849545 8754 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260620 02:53:04.853138 8769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:04.854661 8754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:53:04.854807 8754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-0/wal
uuid: "6b9e385ed07f46599cb8c337f468777b"
format_stamp: "Formatted at 2026-06-20 02:53:04 on dist-test-slave-8p8c"
server_key: "275de7831ea17e55c725a19bbdefd97b"
server_key_iv: "1deed52e70f89634002cade5199fb5ae"
server_key_version: "encryptionkey@0"
I20260620 02:53:04.854972 8754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:53:04.877609 8754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:53:04.881816 8754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:53:04.882114 8754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:53:04.883205 8754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:53:04.884617 8754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:53:04.884691 8754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:04.884773 8754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:53:04.884816 8754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:04.918963 8754 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45371
I20260620 02:53:04.918998 8881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45371 every 8 connection(s)
I20260620 02:53:04.920292 8754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:53:04.922330 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 8754
I20260620 02:53:04.922580 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:53:04.923316 388 external_mini_cluster.cc:1428] Setting key 0d77cda9348b547fed0f8bb197c5f351
Jun 20 02:53:04 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:53:04.943063 8884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:04.923172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48519 (local address 127.0.97.62:45373)
0620 02:53:04.923641 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:04.923647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:04.924785 (+ 1138us) server_negotiation.cc:407] Connection header received
0620 02:53:04.926690 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:04.926698 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:04.926790 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:04.926912 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:04.928406 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:04.929543 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:04.931251 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:04.931726 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:04.931827 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:04.936518 (+ 4691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:04.936524 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:04.936527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:53:04.936562 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:04.939241 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:04.940326 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:04.940331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:04.940334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:04.940458 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:04.941155 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:04.941159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:04.941161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:04.941440 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:53:04.941614 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:04.942607 (+ 993us) server_negotiation.cc:299] Negotiation successful
0620 02:53:04.942779 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":229,"threads_started":1}
I20260620 02:53:04.944797 8885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:04.923759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.1:48519)
0620 02:53:04.924609 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:04.924668 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:04.926322 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:04.927121 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:04.927130 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:04.927676 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:04.928232 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:04.928249 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:04.929733 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:04.929737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:04.931107 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:04.931120 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:04.932052 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:04.932063 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:04.932450 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:04.933344 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:04.933373 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:04.936191 (+ 2818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:53:04.939396 (+ 3205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:53:04.939407 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:53:04.939429 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:53:04.940125 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:53:04.940591 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:53:04.940599 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:53:04.940605 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:53:04.940865 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:53:04.941595 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:53:04.941606 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:53:04.942131 (+ 525us) client_negotiation.cc:769] Sending connection context
0620 02:53:04.942493 (+ 362us) client_negotiation.cc:240] Negotiation successful
0620 02:53:04.943163 (+ 670us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":376,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:53:04.946609 8882 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45373
I20260620 02:53:04.946923 8882 heartbeater.cc:464] Registering TS with master...
I20260620 02:53:04.947685 8882 heartbeater.cc:511] Master 127.0.97.62:45373 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 02:53:04.950105 8693 ts_manager.cc:194] Registered new tserver with Master: 6b9e385ed07f46599cb8c337f468777b (127.0.97.1:45371)
I20260620 02:53:04.951647 8693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48519
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:53:04.979478 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45373
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:53:05.310392 8888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:53:05.310671 8888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:53:05.310766 8888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:53:05.318100 8888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:53:05.318204 8888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:53:05.318315 8888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:53:05.327814 8888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45373
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:53:05.328997 8888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:53:05.330706 8888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:53:05.346419 8894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:05.346536 8896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:05.347183 8893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:53:05.351418 8888 server_base.cc:1034] running on GCE node
I20260620 02:53:05.352003 8888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:53:05.352909 8888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:53:05.354140 8888 hybrid_clock.cc:648] HybridClock initialized: now 1781923985354089 us; error 74 us; skew 500 ppm
I20260620 02:53:05.354496 8888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:53:05 dist-test-slave-8p8c krb5kdc[8648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:53:05.357602 8888 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:53:05.359180 8888 webserver.cc:460] Webserver started at http://127.0.97.2:46567/ using document root <none> and password file <none>
I20260620 02:53:05.359920 8888 fs_manager.cc:359] Metadata directory not provided
I20260620 02:53:05.360060 8888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:53:05.360350 8888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:53:05.363169 8888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-1/data/instance:
uuid: "2bc8d68bb0234e13b7251485d8f7f5de"
format_stamp: "Formatted at 2026-06-20 02:53:05 on dist-test-slave-8p8c"
server_key: "9bd81e7f08aef0c4a1485d77daaf1ee9"
server_key_iv: "37fe44b8dbde8c48ff7bd2b8bda1f79d"
server_key_version: "encryptionkey@0"
I20260620 02:53:05.363914 8888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2bc8d68bb0234e13b7251485d8f7f5de"
format_stamp: "Formatted at 2026-06-20 02:53:05 on dist-test-slave-8p8c"
server_key: "9bd81e7f08aef0c4a1485d77daaf1ee9"
server_key_iv: "37fe44b8dbde8c48ff7bd2b8bda1f79d"
server_key_version: "encryptionkey@0"
I20260620 02:53:05.368736 8888 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 02:53:05.372494 8903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:05.374058 8888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:53:05.374248 8888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-1/wal
uuid: "2bc8d68bb0234e13b7251485d8f7f5de"
format_stamp: "Formatted at 2026-06-20 02:53:05 on dist-test-slave-8p8c"
server_key: "9bd81e7f08aef0c4a1485d77daaf1ee9"
server_key_iv: "37fe44b8dbde8c48ff7bd2b8bda1f79d"
server_key_version: "encryptionkey@0"
I20260620 02:53:05.374491 8888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:53:05.400277 8888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:53:05.404644 8888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:53:05.404940 8888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:53:05.405921 8888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:53:05.407390 8888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:53:05.407490 8888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:05.407635 8888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:53:05.407706 8888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:05.441376 8888 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45001
I20260620 02:53:05.441419 9015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45001 every 8 connection(s)
I20260620 02:53:05.442749 8888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:53:05.451645 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 8888
I20260620 02:53:05.451875 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:53:05.452497 388 external_mini_cluster.cc:1428] Setting key b1f234552284daee8b62775df08534c3
Jun 20 02:53:05 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:53:05.466189 9018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:05.445823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34591 (local address 127.0.97.62:45373)
0620 02:53:05.446272 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:05.446279 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:05.447277 (+ 998us) server_negotiation.cc:407] Connection header received
0620 02:53:05.449143 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:05.449149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:05.449249 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:05.449368 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:05.451041 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:05.451763 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:05.454056 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:05.454713 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:05.454814 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:05.459381 (+ 4567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:05.459391 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:05.459399 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:53:05.459449 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:05.462699 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:05.463734 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:05.463738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:05.463740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:05.463879 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:05.464411 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:05.464414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:05.464416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:05.464701 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:53:05.464814 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:05.465727 (+ 913us) server_negotiation.cc:299] Negotiation successful
0620 02:53:05.465891 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":263,"threads_started":1}
I20260620 02:53:05.467767 9019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:05.446282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.2:34591)
0620 02:53:05.447100 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:05.447141 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:05.448859 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:05.449614 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:05.449623 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:05.450160 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:05.450877 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:05.450896 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:05.451961 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:05.451969 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:05.453882 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:05.453901 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:05.454945 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:05.454954 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:05.455250 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:05.456027 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:05.456056 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:05.459097 (+ 3041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:53:05.462846 (+ 3749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:53:05.462859 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:53:05.462881 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:53:05.463580 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:53:05.463994 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:53:05.464001 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:53:05.464008 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:53:05.464267 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:53:05.464830 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:53:05.464842 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:53:05.465324 (+ 482us) client_negotiation.cc:769] Sending connection context
0620 02:53:05.465628 (+ 304us) client_negotiation.cc:240] Negotiation successful
0620 02:53:05.466219 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":300,"threads_started":1}
I20260620 02:53:05.469331 9016 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45373
I20260620 02:53:05.469671 9016 heartbeater.cc:464] Registering TS with master...
I20260620 02:53:05.470553 9016 heartbeater.cc:511] Master 127.0.97.62:45373 requested a full tablet report, sending...
I20260620 02:53:05.472285 8693 ts_manager.cc:194] Registered new tserver with Master: 2bc8d68bb0234e13b7251485d8f7f5de (127.0.97.2:45001)
I20260620 02:53:05.473574 8693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34591
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:53:05.509400 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45373
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:53:05.839154 9022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:53:05.839438 9022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:53:05.839531 9022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:53:05.846836 9022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:53:05.846964 9022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:53:05.847086 9022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:53:05.857512 9022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45373
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:53:05.858975 9022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:53:05.860385 9022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:53:05.874486 9027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:05.874462 9028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:05.874542 9030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:53:05.875164 9022 server_base.cc:1034] running on GCE node
I20260620 02:53:05.875811 9022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:53:05.876763 9022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:53:05.878034 9022 hybrid_clock.cc:648] HybridClock initialized: now 1781923985877959 us; error 68 us; skew 500 ppm
I20260620 02:53:05.878453 9022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:53:05 dist-test-slave-8p8c krb5kdc[8648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:53:05.881752 9022 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:53:05.883638 9022 webserver.cc:460] Webserver started at http://127.0.97.3:35521/ using document root <none> and password file <none>
I20260620 02:53:05.884362 9022 fs_manager.cc:359] Metadata directory not provided
I20260620 02:53:05.884465 9022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:53:05.884716 9022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:53:05.887814 9022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-2/data/instance:
uuid: "1c0c18453d214e368d254a930d10978e"
format_stamp: "Formatted at 2026-06-20 02:53:05 on dist-test-slave-8p8c"
server_key: "82786b8c67c7ae22803e8619606a3ba1"
server_key_iv: "f4076d5cde67713a6334e3a13507a68e"
server_key_version: "encryptionkey@0"
I20260620 02:53:05.888626 9022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1c0c18453d214e368d254a930d10978e"
format_stamp: "Formatted at 2026-06-20 02:53:05 on dist-test-slave-8p8c"
server_key: "82786b8c67c7ae22803e8619606a3ba1"
server_key_iv: "f4076d5cde67713a6334e3a13507a68e"
server_key_version: "encryptionkey@0"
I20260620 02:53:05.894021 9022 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:53:05.897795 9037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:05.899289 9022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:53:05.899426 9022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-2/wal
uuid: "1c0c18453d214e368d254a930d10978e"
format_stamp: "Formatted at 2026-06-20 02:53:05 on dist-test-slave-8p8c"
server_key: "82786b8c67c7ae22803e8619606a3ba1"
server_key_iv: "f4076d5cde67713a6334e3a13507a68e"
server_key_version: "encryptionkey@0"
I20260620 02:53:05.899600 9022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:53:05.921548 9022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:53:05.926187 9022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:53:05.926574 9022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:53:05.927500 9022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:53:05.928759 9022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:53:05.928833 9022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:05.928926 9022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:53:05.928970 9022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:05.955842 8882 heartbeater.cc:503] Master 127.0.97.62:45373 was elected leader, sending a full tablet report...
I20260620 02:53:05.964780 9022 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41687
I20260620 02:53:05.965054 9149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41687 every 8 connection(s)
I20260620 02:53:05.966084 9022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:53:05.971729 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 9022
I20260620 02:53:05.971907 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:53:05.972456 388 external_mini_cluster.cc:1428] Setting key a85241a64ded8408aa14ac334a40118b
Jun 20 02:53:05 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:53:05.990954 9152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:05.968829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52477 (local address 127.0.97.62:45373)
0620 02:53:05.969338 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:05.969345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:05.970519 (+ 1174us) server_negotiation.cc:407] Connection header received
0620 02:53:05.973290 (+ 2771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:05.973298 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:05.973375 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:05.973507 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:05.975556 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:05.976736 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:05.979123 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:05.979675 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:05.979784 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:05.983988 (+ 4204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:05.983994 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:05.983998 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:53:05.984036 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:05.987584 (+ 3548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:05.988537 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:05.988542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:05.988566 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:05.988717 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:05.989242 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:05.989246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:05.989248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:05.989564 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:53:05.989682 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:05.990524 (+ 842us) server_negotiation.cc:299] Negotiation successful
0620 02:53:05.990693 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":261,"threads_started":1}
I20260620 02:53:05.991973 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:05.969427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.3:52477)
0620 02:53:05.970308 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:05.970358 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:05.972952 (+ 2594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:05.973791 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:05.973803 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:05.974662 (+ 859us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:05.975360 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:05.975379 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:05.976952 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:05.976958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:05.978924 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:05.978940 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:05.979915 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:05.979921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:05.980143 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:05.981071 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:05.981101 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:05.983685 (+ 2584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:53:05.987734 (+ 4049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:53:05.987746 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:53:05.987765 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:53:05.988378 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:53:05.988870 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:53:05.988875 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:53:05.988878 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:53:05.989101 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:53:05.989709 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:53:05.989716 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:53:05.990073 (+ 357us) client_negotiation.cc:769] Sending connection context
0620 02:53:05.990325 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 02:53:05.990827 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":274,"threads_started":1}
I20260620 02:53:05.993458 9150 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45373
I20260620 02:53:05.993786 9150 heartbeater.cc:464] Registering TS with master...
I20260620 02:53:05.994575 9150 heartbeater.cc:511] Master 127.0.97.62:45373 requested a full tablet report, sending...
I20260620 02:53:05.996127 8693 ts_manager.cc:194] Registered new tserver with Master: 1c0c18453d214e368d254a930d10978e (127.0.97.3:41687)
I20260620 02:53:05.996861 8693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52477
I20260620 02:53:06.001390 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260620 02:53:06.151238 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 8754
Jun 20 02:53:06 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:53:06.168947 9161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:06.152661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49212 (local address 127.0.97.1:45371)
0620 02:53:06.153365 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:06.153372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:06.153400 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:53:06.153471 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:06.153478 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:06.153764 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:06.153905 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:06.155354 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:06.156400 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:06.157675 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:06.158236 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:06.158492 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:06.162884 (+ 4392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:06.162897 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:06.162912 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:53:06.162988 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:06.166192 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:06.166992 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:06.166998 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:06.167001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:06.167170 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:06.167623 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:06.167627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:06.167629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:06.167963 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:53:06.168201 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:06.168477 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 02:53:06.168669 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":362,"threads_started":1}
I20260620 02:53:06.170615 8877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:53:06.258548 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 8754
I20260620 02:53:06.275763 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 8888
Jun 20 02:53:06 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:53:06.292604 9163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:06.276861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36246 (local address 127.0.97.2:45001)
0620 02:53:06.277437 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:06.277444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:06.277471 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:53:06.277542 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:06.277548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:06.277777 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:06.277911 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:06.279376 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:06.280391 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:06.281703 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:06.282247 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:06.282347 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:06.286741 (+ 4394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:06.286752 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:06.286768 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:53:06.286810 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:06.289699 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:06.290706 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:06.290713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:06.290715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:06.290842 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:06.291218 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:06.291221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:06.291223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:06.291548 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:53:06.291768 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:06.292105 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 02:53:06.292304 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":306,"threads_started":1}
I20260620 02:53:06.294080 9011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:53:06.379195 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 8888
I20260620 02:53:06.395174 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 9022
Jun 20 02:53:06 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:53:06.411580 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:06.396022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49776 (local address 127.0.97.3:41687)
0620 02:53:06.396641 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:06.396648 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:06.396678 (+ 30us) server_negotiation.cc:407] Connection header received
0620 02:53:06.396741 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:06.396747 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:06.396981 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:06.397169 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:06.398431 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:06.399445 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:06.400698 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:06.401269 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:06.401381 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:06.405913 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:06.405924 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:06.405940 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:53:06.405976 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:06.408930 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:06.409643 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:06.409650 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:06.409653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:06.409794 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:06.410120 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:06.410123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:06.410126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:06.410559 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:53:06.410779 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:06.411050 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 02:53:06.411246 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":313,"threads_started":1}
I20260620 02:53:06.412879 9145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:53:06.499718 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 9022
I20260620 02:53:06.515605 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 8661
I20260620 02:53:06.516816 8723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:53:06.617461 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 8661
I20260620 02:53:06.637573 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45373
--webserver_interface=127.0.97.62
--webserver_port=38185
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:53:06.976756 9168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:53:06.977173 9168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:53:06.977363 9168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:53:06.984856 9168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:53:06.984977 9168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:53:06.985074 9168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:53:06.985164 9168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:53:06.985234 9168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:53:06.995235 9168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=38185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:53:06.996603 9168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:53:06.998071 9168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:53:07.010727 9174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:07.010715 9173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:07.011026 9176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:53:07.011552 9168 server_base.cc:1034] running on GCE node
I20260620 02:53:07.012470 9168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:53:07.014005 9168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:53:07.015254 9168 hybrid_clock.cc:648] HybridClock initialized: now 1781923987015177 us; error 78 us; skew 500 ppm
I20260620 02:53:07.015702 9168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:53:07 dist-test-slave-8p8c krb5kdc[8648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:53:07.019095 9168 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:53:07.020681 9168 webserver.cc:460] Webserver started at http://127.0.97.62:38185/ using document root <none> and password file <none>
I20260620 02:53:07.021413 9168 fs_manager.cc:359] Metadata directory not provided
I20260620 02:53:07.021543 9168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:53:07.027390 9168 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:53:07.030592 9183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:07.032115 9168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 02:53:07.032320 9168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "b5002715c3ff480796ba9a8f7a239be6"
format_stamp: "Formatted at 2026-06-20 02:53:04 on dist-test-slave-8p8c"
server_key: "50617a24533db6e5b15fc35e8fab94c0"
server_key_iv: "169afd27b0b38a90653b75f1746322bf"
server_key_version: "encryptionkey@0"
I20260620 02:53:07.033305 9168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:53:07.058390 9168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:53:07.062680 9168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:53:07.062947 9168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:53:07.081103 9168 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45373
I20260620 02:53:07.081130 9234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45373 every 8 connection(s)
I20260620 02:53:07.082646 9168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:53:07.089406 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 9168
I20260620 02:53:07.089823 9235 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:53:07.090236 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45371
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45373
--webserver_port=43557
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:53:07.093443 9235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6: Bootstrap starting.
I20260620 02:53:07.113523 9235 log.cc:824] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6: Log is configured to *not* fsync() on all Append() calls
I20260620 02:53:07.122741 9235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:53:07.123302 9235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6: Bootstrap complete.
I20260620 02:53:07.133065 9235 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5002715c3ff480796ba9a8f7a239be6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45373 } }
I20260620 02:53:07.133956 9235 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5002715c3ff480796ba9a8f7a239be6, State: Initialized, Role: FOLLOWER
I20260620 02:53:07.134683 9235 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5002715c3ff480796ba9a8f7a239be6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45373 } }
I20260620 02:53:07.134935 9235 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:53:07.135064 9235 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:53:07.135272 9235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:53:07.139608 9235 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5002715c3ff480796ba9a8f7a239be6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45373 } }
I20260620 02:53:07.140162 9235 leader_election.cc:302] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5002715c3ff480796ba9a8f7a239be6; no voters:
I20260620 02:53:07.140852 9235 leader_election.cc:288] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 02:53:07.141000 9239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 02:53:07.142098 9239 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [term 2 LEADER]: Becoming Leader. State: Replica: b5002715c3ff480796ba9a8f7a239be6, State: Running, Role: LEADER
I20260620 02:53:07.142724 9239 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5002715c3ff480796ba9a8f7a239be6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45373 } }
I20260620 02:53:07.143239 9235 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:53:07.147809 9240 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b5002715c3ff480796ba9a8f7a239be6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5002715c3ff480796ba9a8f7a239be6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45373 } } }
I20260620 02:53:07.148190 9240 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [sys.catalog]: This master's current role is: LEADER
I20260620 02:53:07.148602 9241 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5002715c3ff480796ba9a8f7a239be6. Latest consensus state: current_term: 2 leader_uuid: "b5002715c3ff480796ba9a8f7a239be6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5002715c3ff480796ba9a8f7a239be6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45373 } } }
I20260620 02:53:07.148885 9241 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6 [sys.catalog]: This master's current role is: LEADER
I20260620 02:53:07.150878 9249 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:53:07.157763 9249 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:53:07.159494 9249 catalog_manager.cc:1259] Loaded cluster ID: 44d1789f4ba542fda3664099ea0223b1
I20260620 02:53:07.159626 9249 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:53:07.164719 9249 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:53:07.164695 9256 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 02:53:07.166234 9249 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b5002715c3ff480796ba9a8f7a239be6: Loaded TSK: 0
I20260620 02:53:07.167438 9249 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 02:53:07.450124 9237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:53:07.450469 9237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:53:07.450635 9237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:53:07.458082 9237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:53:07.458256 9237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:53:07.458478 9237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:53:07.468366 9237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45371
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=43557
--tserver_master_addrs=127.0.97.62:45373
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:53:07.469702 9237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:53:07.471169 9237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:53:07.485800 9262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:07.485925 9261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:07.486500 9264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:53:07.487186 9237 server_base.cc:1034] running on GCE node
I20260620 02:53:07.487958 9237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:53:07.488993 9237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:53:07.490257 9237 hybrid_clock.cc:648] HybridClock initialized: now 1781923987490191 us; error 47 us; skew 500 ppm
I20260620 02:53:07.490690 9237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:53:07 dist-test-slave-8p8c krb5kdc[8648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923987, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:53:07.494159 9237 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 02:53:07.495918 9237 webserver.cc:460] Webserver started at http://127.0.97.1:43557/ using document root <none> and password file <none>
I20260620 02:53:07.496692 9237 fs_manager.cc:359] Metadata directory not provided
I20260620 02:53:07.496857 9237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:53:07.503126 9237 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.002s
I20260620 02:53:07.506215 9271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:07.507916 9237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 02:53:07.508103 9237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-0/wal
uuid: "6b9e385ed07f46599cb8c337f468777b"
format_stamp: "Formatted at 2026-06-20 02:53:04 on dist-test-slave-8p8c"
server_key: "275de7831ea17e55c725a19bbdefd97b"
server_key_iv: "1deed52e70f89634002cade5199fb5ae"
server_key_version: "encryptionkey@0"
I20260620 02:53:07.509066 9237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:53:07.529506 9237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:53:07.534212 9237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:53:07.534531 9237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:53:07.535557 9237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:53:07.536983 9237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:53:07.537071 9237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:07.537156 9237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:53:07.537218 9237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:07.572393 9237 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45371
I20260620 02:53:07.572516 9383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45371 every 8 connection(s)
I20260620 02:53:07.573932 9237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:53:07.582616 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 9237
I20260620 02:53:07.583207 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:45001
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45373
--webserver_port=46567
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:53:07 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:07 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:07.593344 9387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:07.577086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.1:60923)
0620 02:53:07.577978 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:07.578027 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:07.579785 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:07.581439 (+ 1654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:07.581450 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:07.582179 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:07.582898 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:07.582918 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:07.584588 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:07.584600 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:07.585838 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:07.585850 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:07.587019 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:07.587026 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:07.587308 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:07.588060 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:07.588085 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:07.591981 (+ 3896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":369,"threads_started":1}
W20260620 02:53:07.593362 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:07.577018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60923 (local address 127.0.97.62:45373)
0620 02:53:07.577873 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:07.577891 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:07.578151 (+ 260us) server_negotiation.cc:407] Connection header received
0620 02:53:07.580485 (+ 2334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:07.580500 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:07.580878 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:07.581289 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:07.583097 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:07.584417 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:07.585998 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:07.586771 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:07.586978 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:07.592350 (+ 5372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60923: BlockingRecv error: recv got EOF from 127.0.97.1:60923 (error 108)
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":340,"threads_started":1}
W20260620 02:53:07.594058 9384 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45373 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:45373: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 20 02:53:07 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:07 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:07.602813 9387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:07.594758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.1:59261)
0620 02:53:07.594923 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:07.594943 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:07.595060 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:07.595411 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:07.595415 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:07.595767 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:07.595894 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:07.595902 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:07.596883 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:07.596889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:07.597871 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:07.597881 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:07.598609 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:07.598615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:07.598789 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:07.599419 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:07.599435 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:07.602503 (+ 3068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 02:53:07.602895 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:07.594829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59261 (local address 127.0.97.62:45373)
0620 02:53:07.594983 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:07.594989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:07.595007 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:53:07.595210 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:07.595215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:07.595290 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:07.595390 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:07.596026 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:07.596749 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:07.597996 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:07.598469 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:07.598605 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:07.602678 (+ 4073us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59261: BlockingRecv error: recv got EOF from 127.0.97.1:59261 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 02:53:07 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:07 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:07.612162 9387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:07.603697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.1:47291)
0620 02:53:07.603891 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:07.603912 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:07.604057 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:07.604510 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:07.604514 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:07.604822 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:07.605000 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:07.605008 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:07.606020 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:07.606025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:07.607042 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:07.607053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:07.607703 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:07.607709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:07.607876 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:07.608517 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:07.608535 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:07.611825 (+ 3290us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 02:53:07.612309 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:07.603896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47291 (local address 127.0.97.62:45373)
0620 02:53:07.604110 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:07.604117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:07.604139 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:53:07.604198 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:07.604205 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:07.604307 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:07.604436 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:07.605150 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:07.605879 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:07.607173 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:07.607562 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:07.607693 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:07.612023 (+ 4330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:47291: BlockingRecv error: recv got EOF from 127.0.97.1:47291 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 02:53:07.612660 9384 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 02:53:07.921139 9388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:53:07.921501 9388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:53:07.921680 9388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:53:07.929077 9388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:53:07.929263 9388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:53:07.929471 9388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:53:07.940601 9388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:45001
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=46567
--tserver_master_addrs=127.0.97.62:45373
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:53:07.942144 9388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:53:07.943763 9388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:53:07.960085 9393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:07.960114 9394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:07.960539 9396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:53:07.961223 9388 server_base.cc:1034] running on GCE node
I20260620 02:53:07.962031 9388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:53:07.963171 9388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:53:07.964424 9388 hybrid_clock.cc:648] HybridClock initialized: now 1781923987964331 us; error 82 us; skew 500 ppm
I20260620 02:53:07.964785 9388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:53:07 dist-test-slave-8p8c krb5kdc[8648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923987, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:53:07.968191 9388 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 02:53:07.969740 9388 webserver.cc:460] Webserver started at http://127.0.97.2:46567/ using document root <none> and password file <none>
I20260620 02:53:07.970475 9388 fs_manager.cc:359] Metadata directory not provided
I20260620 02:53:07.970594 9388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:53:07.976173 9388 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:53:07.979349 9403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:07.981020 9388 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:53:07.981215 9388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-1/wal
uuid: "2bc8d68bb0234e13b7251485d8f7f5de"
format_stamp: "Formatted at 2026-06-20 02:53:05 on dist-test-slave-8p8c"
server_key: "9bd81e7f08aef0c4a1485d77daaf1ee9"
server_key_iv: "37fe44b8dbde8c48ff7bd2b8bda1f79d"
server_key_version: "encryptionkey@0"
I20260620 02:53:07.982219 9388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:53:08.013573 9388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:53:08.018158 9388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:53:08.018523 9388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:53:08.019567 9388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:53:08.020911 9388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:53:08.021050 9388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:08.021188 9388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:53:08.021275 9388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:08.056849 9388 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45001
I20260620 02:53:08.056973 9515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45001 every 8 connection(s)
I20260620 02:53:08.058341 9388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:53:08.066900 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 9388
I20260620 02:53:08.067708 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:41687
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45373
--webserver_port=35521
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:53:08 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:08 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 02:53:08.076977 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:08.061118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32853 (local address 127.0.97.62:45373)
0620 02:53:08.061435 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:08.061442 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:08.062805 (+ 1363us) server_negotiation.cc:407] Connection header received
0620 02:53:08.064693 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:08.064697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:08.064759 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:08.064849 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:08.066522 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:08.067384 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:08.070093 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:08.070603 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:08.070707 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:08.076793 (+ 6086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:32853: BlockingRecv error: recv got EOF from 127.0.97.2:32853 (error 108)
Metrics: {"server-negotiator.queue_time_us":180}
I20260620 02:53:08.077968 9518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:08.061757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.2:32853)
0620 02:53:08.062631 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:08.062679 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:08.064421 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:08.065116 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:08.065125 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:08.065671 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:08.066260 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:08.066278 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:08.067566 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:08.067572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:08.069915 (+ 2343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:08.069934 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:08.070839 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:08.070848 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:08.071195 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:08.071979 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:08.072018 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:08.076630 (+ 4612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":297,"threads_started":1}
W20260620 02:53:08.078768 9516 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45373 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:45373: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 20 02:53:08 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:08 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:08.090444 9518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:08.079382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.2:40925)
0620 02:53:08.079640 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:08.079660 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:08.079783 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:08.080109 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:08.080117 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:08.080546 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:08.080809 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:08.080822 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:08.082348 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:08.082357 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:08.084077 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:08.084093 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:08.085148 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:08.085156 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:08.085374 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:08.086107 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:08.086136 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:08.090058 (+ 3922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124}
W20260620 02:53:08.090608 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:08.079474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40925 (local address 127.0.97.62:45373)
0620 02:53:08.079646 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:08.079651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:08.079792 (+ 141us) server_negotiation.cc:407] Connection header received
0620 02:53:08.079882 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:08.079886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:08.079952 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:08.080033 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:08.080964 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:08.082187 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:08.084241 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:08.084954 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:08.085119 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:08.090306 (+ 5187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:40925: BlockingRecv error: recv got EOF from 127.0.97.2:40925 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 02:53:08 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:08 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:08.100719 9518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:08.091436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.2:41033)
0620 02:53:08.091709 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:08.091730 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:08.091846 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:08.092215 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:08.092222 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:08.092646 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:08.092859 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:08.092874 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:08.093961 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:08.093967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:08.095472 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:08.095485 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:08.096312 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:08.096318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:08.096484 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:08.097107 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:08.097129 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:08.100372 (+ 3243us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149}
W20260620 02:53:08.100739 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:08.091516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41033 (local address 127.0.97.62:45373)
0620 02:53:08.091687 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:08.091693 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:08.091808 (+ 115us) server_negotiation.cc:407] Connection header received
0620 02:53:08.091976 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:08.091983 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:08.092061 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:08.092164 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:08.093018 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:08.093824 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:08.095732 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:08.096180 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:08.096281 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:08.100570 (+ 4289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:41033: BlockingRecv error: recv got EOF from 127.0.97.2:41033 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 02:53:08.101055 9516 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 02:53:08.395079 9519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:53:08.395360 9519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:53:08.395529 9519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:53:08.402576 9519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:53:08.402671 9519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:53:08.402877 9519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:53:08.412434 9519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:41687
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=35521
--tserver_master_addrs=127.0.97.62:45373
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:53:08.413782 9519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:53:08.415086 9519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:53:08.428105 9525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:08.428136 9524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:08.428848 9527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:53:08.429880 9519 server_base.cc:1034] running on GCE node
I20260620 02:53:08.430747 9519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:53:08.431931 9519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:53:08.435303 9519 hybrid_clock.cc:648] HybridClock initialized: now 1781923988435250 us; error 37 us; skew 500 ppm
I20260620 02:53:08.435702 9519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:53:08 dist-test-slave-8p8c krb5kdc[8648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781923988, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:53:08.438827 9519 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 02:53:08.440431 9519 webserver.cc:460] Webserver started at http://127.0.97.3:35521/ using document root <none> and password file <none>
I20260620 02:53:08.441123 9519 fs_manager.cc:359] Metadata directory not provided
I20260620 02:53:08.441277 9519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:53:08.446781 9519 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:53:08.449759 9534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:08.451129 9519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:53:08.451303 9519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-2/wal
uuid: "1c0c18453d214e368d254a930d10978e"
format_stamp: "Formatted at 2026-06-20 02:53:05 on dist-test-slave-8p8c"
server_key: "82786b8c67c7ae22803e8619606a3ba1"
server_key_iv: "f4076d5cde67713a6334e3a13507a68e"
server_key_version: "encryptionkey@0"
I20260620 02:53:08.452253 9519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:53:08.481102 9519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:53:08.485476 9519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:53:08.485791 9519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:53:08.486917 9519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:53:08.488258 9519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:53:08.488385 9519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:08.488516 9519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:53:08.488610 9519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:08.522404 9519 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41687
I20260620 02:53:08.522468 9646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41687 every 8 connection(s)
I20260620 02:53:08.523834 9519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:53:08.533530 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 9519
Jun 20 02:53:08 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:08 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 02:53:08.541478 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:08.526663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53497 (local address 127.0.97.62:45373)
0620 02:53:08.526885 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:08.526891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:08.528115 (+ 1224us) server_negotiation.cc:407] Connection header received
0620 02:53:08.530020 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:08.530025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:08.530108 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:08.530215 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:08.531869 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:08.532575 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:08.534200 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:08.534933 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:08.535101 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:08.541176 (+ 6075us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:53497: BlockingRecv error: recv got EOF from 127.0.97.3:53497 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 02:53:08.542306 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:08.527150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.3:53497)
0620 02:53:08.527941 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:08.527986 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:08.529718 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:08.530552 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:08.530562 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:08.531130 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:08.531697 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:08.531715 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:08.532714 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:08.532717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:08.534056 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:08.534068 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:08.535309 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:08.535319 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:08.535650 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:08.536450 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:08.536484 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:08.540958 (+ 4474us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":290,"threads_started":1}
W20260620 02:53:08.543119 9647 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45373 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:45373: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 20 02:53:08 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:08 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:08.552022 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:08.534633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38814 (local address 127.0.97.62:45373)
0620 02:53:08.535276 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:08.535280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:08.535293 (+ 13us) server_negotiation.cc:407] Connection header received
0620 02:53:08.535382 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:08.535386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:08.535436 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:08.535516 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:08.537548 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:08.538613 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:08.539969 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:08.540707 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:08.540841 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:08.544422 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:08.544430 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:08.544446 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:53:08.544484 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:08.548281 (+ 3797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:08.549272 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:08.549282 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:08.549291 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:08.549531 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:08.549894 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:08.549899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:08.549903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:08.550602 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:53:08.550826 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:08.551334 (+ 508us) server_negotiation.cc:299] Negotiation successful
0620 02:53:08.551645 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":411,"threads_started":1}
I20260620 02:53:08.552968 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:08.543758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.3:57625)
0620 02:53:08.544086 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:08.544106 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:08.544209 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:08.544576 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:08.544581 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:08.544967 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:08.545149 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:08.545162 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:08.546249 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:08.546254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:08.547720 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:08.547734 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:08.548593 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:08.548597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:08.548758 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:08.549319 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:08.549344 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:08.552696 (+ 3352us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":70}
W20260620 02:53:08.553092 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:08.543805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57625 (local address 127.0.97.62:45373)
0620 02:53:08.543949 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:08.543954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:08.544156 (+ 202us) server_negotiation.cc:407] Connection header received
0620 02:53:08.544369 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:08.544373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:08.544442 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:08.544519 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:08.545310 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:08.546107 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:08.547867 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:08.548323 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:08.548592 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:08.552859 (+ 4267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:57625: BlockingRecv error: recv got EOF from 127.0.97.3:57625 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 20 02:53:08 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:08 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:08.562000 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:08.553776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.3:58383)
0620 02:53:08.553926 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:08.553947 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:08.554043 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:08.554452 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:08.554457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:08.554781 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:08.554926 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:08.555084 (+ 158us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:08.556521 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:08.556525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:08.557456 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:08.557466 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:08.558321 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:08.558325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:08.558527 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:08.559171 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:08.559188 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:08.561725 (+ 2537us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 02:53:08.562085 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:08.553866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58383 (local address 127.0.97.62:45373)
0620 02:53:08.554113 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:08.554118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:08.554134 (+ 16us) server_negotiation.cc:407] Connection header received
0620 02:53:08.554174 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:08.554178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:08.554247 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:08.554331 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:08.555268 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:08.556398 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:08.557591 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:08.558002 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:08.558151 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:08.561895 (+ 3744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58383: BlockingRecv error: recv got EOF from 127.0.97.3:58383 (error 108)
Metrics: {"server-negotiator.queue_time_us":129}
W20260620 02:53:08.562412 9647 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 02:53:08 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:08 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:08.621764 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:08.613391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.1:47535)
0620 02:53:08.613886 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:08.613902 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:08.614017 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:08.614289 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:08.614292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:08.614797 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:08.614999 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:08.615007 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:08.616254 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:08.616260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:08.617298 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:08.617308 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:08.618007 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:08.618013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:08.618180 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:08.618751 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:08.618767 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:08.621456 (+ 2689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":243,"threads_started":1}
W20260620 02:53:08.621809 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:08.613438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47535 (local address 127.0.97.62:45373)
0620 02:53:08.613681 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:08.613686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:08.613991 (+ 305us) server_negotiation.cc:407] Connection header received
0620 02:53:08.614103 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:08.614107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:08.614166 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:08.614246 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:08.615165 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:08.616118 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:08.617437 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:08.617840 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:08.617976 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:08.621654 (+ 3678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:47535: BlockingRecv error: recv got EOF from 127.0.97.1:47535 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 02:53:09 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:09 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:09.109812 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:09.101597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.2:42637)
0620 02:53:09.102138 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:09.102153 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:09.102308 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:09.102678 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:09.102682 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:09.103161 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:09.103336 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:09.103344 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:09.104328 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:09.104333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:09.105281 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:09.105292 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:09.106011 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:09.106016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:09.106179 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:09.106802 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:09.106818 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:09.109479 (+ 2661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":298,"threads_started":1}
W20260620 02:53:09.109906 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:09.101717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42637 (local address 127.0.97.62:45373)
0620 02:53:09.101974 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:09.101982 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:09.102237 (+ 255us) server_negotiation.cc:407] Connection header received
0620 02:53:09.102462 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:09.102468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:09.102540 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:09.102649 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:09.103475 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:09.104192 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:09.105429 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:09.105833 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:09.105957 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:09.109671 (+ 3714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:42637: BlockingRecv error: recv got EOF from 127.0.97.2:42637 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Jun 20 02:53:09 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:09 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:09.576254 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:09.562992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.3:48965)
0620 02:53:09.563512 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:09.563528 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:09.563682 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:09.564050 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:09.564055 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:09.564550 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:09.564729 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:09.564737 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:09.565731 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:09.565739 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:09.566760 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:09.566772 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:09.567488 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:09.567494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:09.567655 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:09.572904 (+ 5249us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:09.572928 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:09.575903 (+ 2975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":274,"threads_started":1}
W20260620 02:53:09.576387 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:09.563050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48965 (local address 127.0.97.62:45373)
0620 02:53:09.563282 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:09.563290 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:09.563611 (+ 321us) server_negotiation.cc:407] Connection header received
0620 02:53:09.563818 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:09.563822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:09.563881 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:09.564027 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:09.564867 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:09.565588 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:09.566897 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:09.567305 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:09.567439 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:09.576123 (+ 8684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:48965: BlockingRecv error: recv got EOF from 127.0.97.3:48965 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 02:53:09 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:09 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:09.631433 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:09.622790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.1:40833)
0620 02:53:09.623297 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:09.623314 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:09.623437 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:09.623736 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:09.623739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:09.624133 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:09.624278 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:09.624287 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:09.625443 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:09.625451 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:09.626930 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:09.626944 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:09.627661 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:09.627665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:09.627774 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:09.628326 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:09.628343 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:09.631197 (+ 2854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":307,"threads_started":1}
W20260620 02:53:09.631771 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:09.622846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40833 (local address 127.0.97.62:45373)
0620 02:53:09.623084 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:09.623094 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:09.623415 (+ 321us) server_negotiation.cc:407] Connection header received
0620 02:53:09.623537 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:09.623541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:09.623616 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:09.623697 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:09.624444 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:09.625274 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:09.627094 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:09.627503 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:09.627609 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:09.631380 (+ 3771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:40833: BlockingRecv error: recv got EOF from 127.0.97.1:40833 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 02:53:10 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:10 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:10.119552 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:10.110777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.2:38307)
0620 02:53:10.111317 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:10.111335 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:10.111461 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:10.111796 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:10.111799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:10.112230 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:10.112392 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:10.112400 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:10.113736 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:10.113740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:10.114698 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:10.114710 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:10.115556 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:10.115561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:10.115724 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:10.116509 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:10.116528 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:10.119322 (+ 2794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":312,"threads_started":1}
W20260620 02:53:10.119836 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:10.110950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38307 (local address 127.0.97.62:45373)
0620 02:53:10.111211 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:10.111217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:10.111444 (+ 227us) server_negotiation.cc:407] Connection header received
0620 02:53:10.111569 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:10.111572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:10.111634 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:10.111719 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:10.112543 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:10.113550 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:10.114909 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:10.115384 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:10.115502 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:10.119591 (+ 4089us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38307: BlockingRecv error: recv got EOF from 127.0.97.2:38307 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
Jun 20 02:53:10 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:10 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:10.586190 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:10.577580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.3:39495)
0620 02:53:10.578468 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:10.578489 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:10.578627 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:10.578983 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:10.578988 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:10.579428 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:10.579568 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:10.579577 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:10.580574 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:10.580583 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:10.581482 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:10.581493 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:10.582183 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:10.582189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:10.582349 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:10.583056 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:10.583073 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:10.585885 (+ 2812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":746,"thread_start_us":605,"threads_started":1}
W20260620 02:53:10.586378 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:10.578220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39495 (local address 127.0.97.62:45373)
0620 02:53:10.578420 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:10.578426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:10.578582 (+ 156us) server_negotiation.cc:407] Connection header received
0620 02:53:10.578766 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:10.578770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:10.578834 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:10.578951 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:10.579718 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:10.580431 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:10.581619 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:10.582034 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:10.582167 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:10.586124 (+ 3957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:39495: BlockingRecv error: recv got EOF from 127.0.97.3:39495 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 02:53:10 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:10 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:10.640942 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:10.632386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.1:38809)
0620 02:53:10.632959 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:10.632976 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:10.633132 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:10.633487 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:10.633491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:10.633859 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:10.634019 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:10.634028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:10.635112 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:10.635117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:10.636035 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:10.636051 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:10.636743 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:10.636748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:10.636928 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:10.637515 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:10.637532 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:10.640571 (+ 3039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":315,"threads_started":1}
W20260620 02:53:10.641023 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:10.632477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38809 (local address 127.0.97.62:45373)
0620 02:53:10.632758 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:10.632767 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:10.633077 (+ 310us) server_negotiation.cc:407] Connection header received
0620 02:53:10.633275 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:10.633280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:10.633354 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:10.633458 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:10.634176 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:10.634977 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:10.636185 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:10.636603 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:10.636733 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:10.640775 (+ 4042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:38809: BlockingRecv error: recv got EOF from 127.0.97.1:38809 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Jun 20 02:53:11 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:11 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:11.129158 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:11.120498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.2:51505)
0620 02:53:11.121020 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:11.121044 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:11.121152 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:11.121416 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:11.121419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:11.121742 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:11.121906 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:11.121914 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:11.123001 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:11.123005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:11.124036 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:11.124050 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:11.124771 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:11.124775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:11.124946 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:11.125579 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:11.125597 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:11.128768 (+ 3171us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":280,"threads_started":1}
W20260620 02:53:11.129195 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:11.120607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51505 (local address 127.0.97.62:45373)
0620 02:53:11.120801 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:11.120806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:11.121144 (+ 338us) server_negotiation.cc:407] Connection header received
0620 02:53:11.121250 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:11.121254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:11.121315 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:11.121406 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:11.122048 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:11.122839 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:11.124188 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:11.124613 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:11.124715 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:11.128962 (+ 4247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51505: BlockingRecv error: recv got EOF from 127.0.97.2:51505 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 20 02:53:11 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:11 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:11.600770 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:11.587167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.3:38395)
0620 02:53:11.587707 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:11.587722 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:11.587835 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:11.588171 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:11.588175 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:11.588521 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:11.588689 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:11.588697 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:11.589808 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:11.589818 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:11.590767 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:11.590779 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:11.591484 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:11.591491 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:11.591671 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:11.597606 (+ 5935us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:11.597626 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:11.600379 (+ 2753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":299,"threads_started":1}
W20260620 02:53:11.600859 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:11.587323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38395 (local address 127.0.97.62:45373)
0620 02:53:11.587516 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:11.587521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:11.587800 (+ 279us) server_negotiation.cc:407] Connection header received
0620 02:53:11.587935 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:11.587938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:11.588006 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:11.588093 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:11.588836 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:11.589658 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:11.590905 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:11.591337 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:11.591482 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:11.600643 (+ 9161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:38395: BlockingRecv error: recv got EOF from 127.0.97.3:38395 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 02:53:11 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:11 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:11.653002 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:11.641868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.1:45951)
0620 02:53:11.642436 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:11.642457 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:11.642600 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:11.643011 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:11.643017 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:11.643569 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:11.643813 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:11.643823 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:11.645394 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:11.645399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:11.646992 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:11.647006 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:11.648124 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:11.648130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:11.648361 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:11.649182 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:11.649200 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:11.652605 (+ 3405us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":275,"threads_started":1}
W20260620 02:53:11.653414 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:11.642221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45951 (local address 127.0.97.62:45373)
0620 02:53:11.642437 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:11.642443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:11.642522 (+ 79us) server_negotiation.cc:407] Connection header received
0620 02:53:11.642729 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:11.642736 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:11.642818 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:11.642928 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:11.643986 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:11.645253 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:11.647144 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:11.647970 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:11.648105 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:11.653163 (+ 5058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45951: BlockingRecv error: recv got EOF from 127.0.97.1:45951 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 02:53:12 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:12 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:12.139478 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:12.130241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.2:56719)
0620 02:53:12.130749 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:12.130767 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:12.130901 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:12.131243 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:12.131247 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:12.131712 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:12.131933 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:12.131948 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:12.133248 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:12.133252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:12.134675 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:12.134689 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:12.135430 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:12.135443 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:12.135591 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:12.136307 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:12.136325 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:12.139254 (+ 2929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":307,"threads_started":1}
W20260620 02:53:12.139675 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:12.130315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56719 (local address 127.0.97.62:45373)
0620 02:53:12.130551 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:12.130561 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:12.130892 (+ 331us) server_negotiation.cc:407] Connection header received
0620 02:53:12.131005 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:12.131008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:12.131093 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:12.131187 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:12.132213 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:12.133084 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:12.134824 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:12.135273 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:12.135375 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:12.139431 (+ 4056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56719: BlockingRecv error: recv got EOF from 127.0.97.2:56719 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 02:53:12 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:12 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:12.609740 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:12.601848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.3:40291)
0620 02:53:12.602407 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:12.602423 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:12.602555 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:12.603006 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:12.603009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:12.603372 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:12.603511 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:12.603518 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:12.604537 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:12.604541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:12.605504 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:12.605515 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:12.606141 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:12.606145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:12.606261 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:12.606871 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:12.606888 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:12.609519 (+ 2631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":312,"threads_started":1}
W20260620 02:53:12.610047 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:12.601951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40291 (local address 127.0.97.62:45373)
0620 02:53:12.602142 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:12.602146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:12.602521 (+ 375us) server_negotiation.cc:407] Connection header received
0620 02:53:12.602747 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:12.602753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:12.602832 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:12.602947 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:12.603636 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:12.604414 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:12.605634 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:12.606011 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:12.606103 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:12.609737 (+ 3634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40291: BlockingRecv error: recv got EOF from 127.0.97.3:40291 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 02:53:12 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:12 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:12.662667 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:12.654021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.1:52729)
0620 02:53:12.654665 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:12.654682 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:12.654841 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:12.655215 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:12.655221 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:12.655632 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:12.655837 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:12.655848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:12.656860 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:12.656868 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:12.657797 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:12.657807 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:12.658526 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:12.658532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:12.658704 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:12.659333 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:12.659350 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:12.662297 (+ 2947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":337,"threads_started":1}
W20260620 02:53:12.662737 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:12.654208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52729 (local address 127.0.97.62:45373)
0620 02:53:12.654493 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:12.654501 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:12.654768 (+ 267us) server_negotiation.cc:407] Connection header received
0620 02:53:12.654963 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:12.654968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:12.655063 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:12.655169 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:12.655979 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:12.656717 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:12.657939 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:12.658337 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:12.658525 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:12.662530 (+ 4005us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:52729: BlockingRecv error: recv got EOF from 127.0.97.1:52729 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Jun 20 02:53:13 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:13 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:13.149358 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:13.140454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.2:56741)
0620 02:53:13.140989 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:13.141008 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:13.141124 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:13.141426 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:13.141430 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:13.141797 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:13.141964 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:13.141971 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:13.143185 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:13.143189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:13.144257 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:13.144270 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:13.145187 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:13.145194 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:13.145322 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:13.146051 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:13.146077 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:13.149016 (+ 2939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":281,"threads_started":1}
W20260620 02:53:13.149466 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:13.140689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56741 (local address 127.0.97.62:45373)
0620 02:53:13.140906 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:13.140913 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:13.141136 (+ 223us) server_negotiation.cc:407] Connection header received
0620 02:53:13.141204 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:13.141207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:13.141270 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:13.141360 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:13.142132 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:13.143009 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:13.144396 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:13.144991 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:13.145089 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:13.149207 (+ 4118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56741: BlockingRecv error: recv got EOF from 127.0.97.2:56741 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Jun 20 02:53:13 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:13 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:13.619298 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:13.610724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.3:37569)
0620 02:53:13.611340 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:13.611358 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:13.611515 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:13.611889 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:13.611894 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:13.612373 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:13.612554 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:13.612562 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:13.613664 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:13.613668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:13.614635 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:13.614646 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:13.615416 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:13.615420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:13.615558 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:13.616199 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:13.616215 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:13.619062 (+ 2847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":266,"threads_started":1}
W20260620 02:53:13.619510 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:13.610800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37569 (local address 127.0.97.62:45373)
0620 02:53:13.611031 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:13.611040 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:13.611447 (+ 407us) server_negotiation.cc:407] Connection header received
0620 02:53:13.611651 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:13.611655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:13.611711 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:13.611823 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:13.612678 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:13.613561 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:13.614785 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:13.615283 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:13.615387 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:13.619276 (+ 3889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:37569: BlockingRecv error: recv got EOF from 127.0.97.3:37569 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 02:53:13 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:13 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:13.672791 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:13.663730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.1:51077)
0620 02:53:13.664377 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:13.664396 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:13.664557 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:13.664899 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:13.664905 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:13.665299 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:13.665449 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:13.665458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:13.666507 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:13.666514 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:13.667415 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:13.667425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:13.668161 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:13.668170 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:13.668354 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:13.669018 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:13.669038 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:13.672467 (+ 3429us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":416,"threads_started":1}
W20260620 02:53:13.672948 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:13.663823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51077 (local address 127.0.97.62:45373)
0620 02:53:13.664202 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:13.664210 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:13.664486 (+ 276us) server_negotiation.cc:407] Connection header received
0620 02:53:13.664679 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:13.664685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:13.664752 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:13.664856 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:13.665592 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:13.666314 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:13.667560 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:13.668003 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:13.668140 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:13.672658 (+ 4518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:51077: BlockingRecv error: recv got EOF from 127.0.97.1:51077 (error 108)
Metrics: {"server-negotiator.queue_time_us":209}
Jun 20 02:53:14 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:14 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:14.158806 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:14.150283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.2:40097)
0620 02:53:14.150843 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:14.150859 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:14.150965 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:14.151243 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:14.151246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:14.151616 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:14.151779 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:14.151787 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:14.152812 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:14.152816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:14.153692 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:14.153702 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:14.154445 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:14.154450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:14.154580 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:14.155239 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:14.155256 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:14.158602 (+ 3346us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":331,"threads_started":1}
W20260620 02:53:14.159096 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:14.150349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40097 (local address 127.0.97.62:45373)
0620 02:53:14.150580 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:14.150584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:14.150931 (+ 347us) server_negotiation.cc:407] Connection header received
0620 02:53:14.151059 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:14.151062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:14.151121 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:14.151203 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:14.151924 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:14.152657 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:14.153844 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:14.154247 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:14.154349 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:14.158814 (+ 4465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:40097: BlockingRecv error: recv got EOF from 127.0.97.2:40097 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
Jun 20 02:53:14 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:14 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:14.633279 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:14.620293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.3:53175)
0620 02:53:14.620796 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:14.620811 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:14.620922 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:14.621208 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:14.621211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:14.621567 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:14.621737 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:14.621745 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:14.622731 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:14.622735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:14.623655 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:14.623667 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:14.624318 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:14.624322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:14.624441 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:14.629532 (+ 5091us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:14.629553 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:14.632962 (+ 3409us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":301,"threads_started":1}
W20260620 02:53:14.633445 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:14.620395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53175 (local address 127.0.97.62:45373)
0620 02:53:14.620633 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:14.620639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:14.620892 (+ 253us) server_negotiation.cc:407] Connection header received
0620 02:53:14.621018 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:14.621022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:14.621080 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:14.621165 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:14.621853 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:14.622607 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:14.623781 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:14.624193 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:14.624285 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:14.633203 (+ 8918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:53175: BlockingRecv error: recv got EOF from 127.0.97.3:53175 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 02:53:14 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:14 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:14.682439 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:14.673903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.1:55591)
0620 02:53:14.674572 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:14.674589 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:14.674700 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:14.674992 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:14.674995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:14.675369 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:14.675561 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:14.675570 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:14.676642 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:14.676646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:14.677522 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:14.677543 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:14.678231 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:14.678235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:14.678340 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:14.679005 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:14.679022 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:14.682143 (+ 3121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":313,"threads_started":1}
W20260620 02:53:14.682699 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:14.673957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55591 (local address 127.0.97.62:45373)
0620 02:53:14.674201 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:14.674206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:14.674666 (+ 460us) server_negotiation.cc:407] Connection header received
0620 02:53:14.674800 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:14.674804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:14.674866 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:14.674956 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:14.675690 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:14.676514 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:14.677672 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:14.678097 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:14.678195 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:14.682341 (+ 4146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55591: BlockingRecv error: recv got EOF from 127.0.97.1:55591 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Jun 20 02:53:15 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:15 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:15.168231 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:15.159797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.2:51871)
0620 02:53:15.160396 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:15.160412 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:15.160532 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:15.160807 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:15.160810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:15.161257 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:15.161417 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:15.161425 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:15.162497 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:15.162502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:15.163539 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:15.163551 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:15.164294 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:15.164299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:15.164413 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:15.165042 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:15.165060 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:15.168002 (+ 2942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":260,"threads_started":1}
W20260620 02:53:15.168430 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:15.159836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51871 (local address 127.0.97.62:45373)
0620 02:53:15.160115 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:15.160122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:15.160494 (+ 372us) server_negotiation.cc:407] Connection header received
0620 02:53:15.160614 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:15.160617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:15.160676 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:15.160753 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:15.161576 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:15.162300 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:15.163693 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:15.164137 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:15.164238 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:15.168202 (+ 3964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51871: BlockingRecv error: recv got EOF from 127.0.97.2:51871 (error 108)
Metrics: {"server-negotiator.queue_time_us":137}
Jun 20 02:53:15 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:15 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:15.643595 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:15.634279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.3:35247)
0620 02:53:15.635023 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:15.635040 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:15.635208 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:15.635644 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:15.635649 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:15.636293 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:15.636542 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:15.636559 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:15.637815 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:15.637820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:15.638778 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:15.638789 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:15.639515 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:15.639521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:15.639681 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:15.640397 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:15.640416 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:15.643261 (+ 2845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":298,"threads_started":1}
W20260620 02:53:15.643725 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:15.634450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35247 (local address 127.0.97.62:45373)
0620 02:53:15.634666 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:15.634671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:15.635175 (+ 504us) server_negotiation.cc:407] Connection header received
0620 02:53:15.635367 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:15.635371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:15.635465 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:15.635602 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:15.636718 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:15.637667 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:15.638922 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:15.639352 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:15.639495 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:15.643471 (+ 3976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35247: BlockingRecv error: recv got EOF from 127.0.97.3:35247 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 02:53:15 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:15 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:15.692602 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:15.683344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.1:49719)
0620 02:53:15.683886 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:15.683905 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:15.684018 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:15.684381 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:15.684393 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:15.684932 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:15.685130 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:15.685145 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:15.686718 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:15.686722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:15.687644 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:15.687657 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:15.688474 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:15.688480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:15.688603 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:15.689326 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:15.689343 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:15.692369 (+ 3026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":308,"threads_started":1}
W20260620 02:53:15.692894 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:15.683566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49719 (local address 127.0.97.62:45373)
0620 02:53:15.683746 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:15.683751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:15.683979 (+ 228us) server_negotiation.cc:407] Connection header received
0620 02:53:15.684169 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:15.684172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:15.684231 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:15.684315 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:15.685318 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:15.686552 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:15.687788 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:15.688221 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:15.688328 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:15.692612 (+ 4284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49719: BlockingRecv error: recv got EOF from 127.0.97.1:49719 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 02:53:16 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:16 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:16.178000 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:16.169191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.2:46833)
0620 02:53:16.169731 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:16.169746 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:16.169897 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:16.170277 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:16.170281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:16.170753 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:16.170897 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:16.170906 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:16.172059 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:16.172066 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:16.172980 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:16.172991 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:16.173723 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:16.173729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:16.173890 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:16.174588 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:16.174608 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:16.177764 (+ 3156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":274,"threads_started":1}
W20260620 02:53:16.178257 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:16.169340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46833 (local address 127.0.97.62:45373)
0620 02:53:16.169551 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:16.169559 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:16.169828 (+ 269us) server_negotiation.cc:407] Connection header received
0620 02:53:16.170067 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:16.170071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:16.170131 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:16.170246 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:16.171049 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:16.171908 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:16.173117 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:16.173545 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:16.173686 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:16.177969 (+ 4283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46833: BlockingRecv error: recv got EOF from 127.0.97.2:46833 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 02:53:16 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:16 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:16.653127 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:16.644669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.3:40097)
0620 02:53:16.645221 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:16.645243 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:16.645412 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:16.645742 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:16.645745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:16.646121 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:16.646258 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:16.646265 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:16.647494 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:16.647497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:16.648402 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:16.648413 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:16.649302 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:16.649307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:16.649428 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:16.650107 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:16.650124 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:16.652907 (+ 2783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":264,"threads_started":1}
W20260620 02:53:16.653453 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:16.644748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40097 (local address 127.0.97.62:45373)
0620 02:53:16.644952 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:16.644960 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:16.645362 (+ 402us) server_negotiation.cc:407] Connection header received
0620 02:53:16.645550 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:16.645554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:16.645619 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:16.645710 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:16.646479 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:16.647311 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:16.648590 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:16.649115 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:16.649223 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:16.653126 (+ 3903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40097: BlockingRecv error: recv got EOF from 127.0.97.3:40097 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 20 02:53:16 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:16 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:16.702600 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:16.693692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.1:60591)
0620 02:53:16.694348 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:16.694421 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:16.694562 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:16.694853 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:16.694856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:16.695277 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:16.695479 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:16.695489 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:16.696526 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:16.696530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:16.697535 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:16.697547 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:16.698400 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:16.698405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:16.698657 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:16.699242 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:16.699260 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:16.702267 (+ 3007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":413,"threads_started":1}
W20260620 02:53:16.702739 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:16.693739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60591 (local address 127.0.97.62:45373)
0620 02:53:16.693956 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:16.693962 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:16.694508 (+ 546us) server_negotiation.cc:407] Connection header received
0620 02:53:16.694683 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:16.694686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:16.694755 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:16.694832 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:16.695618 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:16.696386 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:16.697682 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:16.698226 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:16.698350 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:16.702517 (+ 4167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60591: BlockingRecv error: recv got EOF from 127.0.97.1:60591 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 02:53:17 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:17 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:17.187884 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:17.179102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.2:36435)
0620 02:53:17.179664 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:17.179686 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:17.179794 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:17.180133 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:17.180136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:17.180508 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:17.180667 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:17.180675 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:17.181673 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:17.181677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:17.182731 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:17.182743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:17.183682 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:17.183688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:17.183870 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:17.184556 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:17.184572 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:17.187645 (+ 3073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":338,"threads_started":1}
W20260620 02:53:17.188063 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:17.179213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36435 (local address 127.0.97.62:45373)
0620 02:53:17.179382 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:17.179386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:17.179761 (+ 375us) server_negotiation.cc:407] Connection header received
0620 02:53:17.179931 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:17.179935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:17.179996 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:17.180085 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:17.180794 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:17.181550 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:17.182865 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:17.183527 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:17.183661 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:17.187827 (+ 4166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:36435: BlockingRecv error: recv got EOF from 127.0.97.2:36435 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 02:53:17 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:17 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:17.662618 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:17.654050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.3:46425)
0620 02:53:17.654624 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:17.654639 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:17.654747 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:17.655114 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:17.655118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:17.655495 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:17.655709 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:17.655719 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:17.656794 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:17.656801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:17.657773 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:17.657785 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:17.658541 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:17.658549 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:17.658732 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:17.659391 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:17.659408 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:17.662246 (+ 2838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":339,"threads_started":1}
W20260620 02:53:17.662708 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:17.654117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46425 (local address 127.0.97.62:45373)
0620 02:53:17.654315 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:17.654320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:17.654725 (+ 405us) server_negotiation.cc:407] Connection header received
0620 02:53:17.654848 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:17.654852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:17.654943 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:17.655041 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:17.655911 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:17.656633 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:17.657922 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:17.658352 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:17.658523 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:17.662484 (+ 3961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:46425: BlockingRecv error: recv got EOF from 127.0.97.3:46425 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 20 02:53:17 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:17 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:17.712107 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:17.703542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.1:59821)
0620 02:53:17.704031 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:17.704078 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:17.704233 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:17.704567 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:17.704573 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:17.704955 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:17.705105 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:17.705114 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:17.706127 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:17.706136 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:17.707148 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:17.707158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:17.707864 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:17.707870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:17.708041 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:17.708727 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:17.708744 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:17.711766 (+ 3022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":276,"threads_started":1}
W20260620 02:53:17.712237 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:17.703605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59821 (local address 127.0.97.62:45373)
0620 02:53:17.703845 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:17.703854 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:17.704168 (+ 314us) server_negotiation.cc:407] Connection header received
0620 02:53:17.704368 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:17.704371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:17.704441 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:17.704541 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:17.705253 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:17.705991 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:17.707285 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:17.707716 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:17.707849 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:17.711971 (+ 4122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59821: BlockingRecv error: recv got EOF from 127.0.97.1:59821 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 02:53:18 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:18 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:18.197412 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:18.188885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.2:55005)
0620 02:53:18.189536 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:18.189553 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:18.189668 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:18.190037 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:18.190041 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:18.190513 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:18.190682 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:18.190690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:18.191779 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:18.191784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:18.192714 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:18.192725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:18.193575 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:18.193579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:18.193697 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:18.194351 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:18.194425 (+ 74us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:18.197174 (+ 2749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":370,"threads_started":1}
W20260620 02:53:18.197788 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:18.188960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55005 (local address 127.0.97.62:45373)
0620 02:53:18.189180 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:18.189185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:18.189643 (+ 458us) server_negotiation.cc:407] Connection header received
0620 02:53:18.189814 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:18.189818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:18.189879 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:18.189974 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:18.190827 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:18.191601 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:18.192865 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:18.193414 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:18.193546 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:18.197412 (+ 3866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55005: BlockingRecv error: recv got EOF from 127.0.97.2:55005 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Jun 20 02:53:18 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:18 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:18.672372 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:18.663699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.3:49631)
0620 02:53:18.664476 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:18.664491 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:18.664603 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:18.664917 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:18.664920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:18.665296 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:18.665438 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:18.665446 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:18.666537 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:18.666541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:18.667471 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:18.667481 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:18.668220 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:18.668226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:18.668363 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:18.669006 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:18.669026 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:18.671986 (+ 2960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":554,"threads_started":1}
W20260620 02:53:18.672441 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:18.663929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49631 (local address 127.0.97.62:45373)
0620 02:53:18.664169 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:18.664178 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:18.664587 (+ 409us) server_negotiation.cc:407] Connection header received
0620 02:53:18.664703 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:18.664706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:18.664780 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:18.664866 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:18.665590 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:18.666304 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:18.667629 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:18.668048 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:18.668149 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:18.672194 (+ 4045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:49631: BlockingRecv error: recv got EOF from 127.0.97.3:49631 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 02:53:18 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:18 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:18.721546 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:18.713125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.1:39863)
0620 02:53:18.713591 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:18.713607 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:18.713718 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:18.714133 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:18.714137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:18.714764 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:18.714973 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:18.714984 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:18.716011 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:18.716015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:18.716925 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:18.716937 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:18.717601 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:18.717604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:18.717717 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:18.718309 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:18.718326 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:18.721203 (+ 2877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":243,"threads_started":1}
W20260620 02:53:18.721621 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:18.713215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39863 (local address 127.0.97.62:45373)
0620 02:53:18.713476 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:18.713483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:18.713688 (+ 205us) server_negotiation.cc:407] Connection header received
0620 02:53:18.713807 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:18.713818 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:18.713890 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:18.713978 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:18.715124 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:18.715836 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:18.717077 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:18.717477 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:18.717575 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:18.721384 (+ 3809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:39863: BlockingRecv error: recv got EOF from 127.0.97.1:39863 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Jun 20 02:53:19 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:19 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:19.206770 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:19.198292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.2:59399)
0620 02:53:19.198837 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:19.198857 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:19.199038 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:19.199447 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:19.199452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:19.200012 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:19.200262 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:19.200274 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:19.201470 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:19.201474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:19.202393 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:19.202404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:19.203054 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:19.203057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:19.203178 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:19.203785 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:19.203802 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:19.206512 (+ 2710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":295,"threads_started":1}
W20260620 02:53:19.206988 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:19.198476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59399 (local address 127.0.97.62:45373)
0620 02:53:19.198682 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:19.198689 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:19.198980 (+ 291us) server_negotiation.cc:407] Connection header received
0620 02:53:19.199201 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:19.199205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:19.199276 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:19.199360 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:19.200450 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:19.201343 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:19.202516 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:19.202931 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:19.203024 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:19.206727 (+ 3703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59399: BlockingRecv error: recv got EOF from 127.0.97.2:59399 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 02:53:19 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:19 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:19.687795 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:19.673536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.3:49139)
0620 02:53:19.674111 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:19.674128 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:19.674289 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:19.674723 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:19.674728 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:19.675247 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:19.675506 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:19.675522 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:19.676568 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:19.676572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:19.677515 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:19.677525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:19.678291 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:19.678295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:19.678455 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:19.684002 (+ 5547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:19.684021 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:19.687483 (+ 3462us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"spinlock_wait_cycles":76672,"thread_start_us":416,"threads_started":1}
W20260620 02:53:19.687980 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:19.673782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49139 (local address 127.0.97.62:45373)
0620 02:53:19.673961 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:19.673966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:19.674219 (+ 253us) server_negotiation.cc:407] Connection header received
0620 02:53:19.674490 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:19.674496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:19.674571 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:19.674659 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:19.675655 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:19.676423 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:19.677670 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:19.678146 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:19.678260 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:19.687730 (+ 9470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:49139: BlockingRecv error: recv got EOF from 127.0.97.3:49139 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 02:53:19 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:19 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:19.731137 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:19.722601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.1:40091)
0620 02:53:19.723136 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:19.723152 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:19.723316 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:19.723687 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:19.723691 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:19.724149 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:19.724297 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:19.724306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:19.725309 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:19.725316 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:19.726236 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:19.726247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:19.727018 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:19.727023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:19.727210 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:19.727848 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:19.727867 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:19.730907 (+ 3040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":289,"threads_started":1}
W20260620 02:53:19.731490 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:19.722685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40091 (local address 127.0.97.62:45373)
0620 02:53:19.722948 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:19.722955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:19.723243 (+ 288us) server_negotiation.cc:407] Connection header received
0620 02:53:19.723443 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:19.723448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:19.723548 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:19.723654 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:19.724438 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:19.725172 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:19.726397 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:19.726874 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:19.727009 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:19.731132 (+ 4123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:40091: BlockingRecv error: recv got EOF from 127.0.97.1:40091 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Jun 20 02:53:20 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:20 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:20.216337 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:20.207667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.2:60729)
0620 02:53:20.208178 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:20.208194 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:20.208311 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:20.208646 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:20.208650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:20.209084 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:20.209229 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:20.209238 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:20.210191 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:20.210195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:20.211253 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:20.211265 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:20.211966 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:20.211971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:20.212095 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:20.212731 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:20.212749 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:20.216004 (+ 3255us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":312,"threads_started":1}
W20260620 02:53:20.216454 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:20.207835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60729 (local address 127.0.97.62:45373)
0620 02:53:20.208136 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:20.208143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:20.208276 (+ 133us) server_negotiation.cc:407] Connection header received
0620 02:53:20.208409 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:20.208412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:20.208471 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:20.208559 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:20.209361 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:20.210051 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:20.211387 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:20.211796 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:20.211891 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:20.216201 (+ 4310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:60729: BlockingRecv error: recv got EOF from 127.0.97.2:60729 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 20 02:53:20 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:20 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:20.698264 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:20.688913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.3:53097)
0620 02:53:20.689494 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:20.689513 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:20.689632 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:20.690004 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:20.690007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:20.690443 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:20.690678 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:20.690690 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:20.692315 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:20.692320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:20.693649 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:20.693662 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:20.694430 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:20.694443 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:20.694597 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:20.695288 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:20.695306 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:20.698037 (+ 2731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":347,"threads_started":1}
W20260620 02:53:20.698630 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:20.688975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53097 (local address 127.0.97.62:45373)
0620 02:53:20.689187 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:20.689191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:20.689598 (+ 407us) server_negotiation.cc:407] Connection header received
0620 02:53:20.689779 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:20.689784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:20.689849 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:20.689941 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:20.691001 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:20.692104 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:20.693796 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:20.694238 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:20.694338 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:20.698296 (+ 3958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:53097: BlockingRecv error: recv got EOF from 127.0.97.3:53097 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Jun 20 02:53:20 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:20 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:20.741700 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:20.732281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.1:50627)
0620 02:53:20.732805 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:20.732821 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:20.732936 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:20.733348 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:20.733352 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:20.733772 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:20.733954 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:20.733963 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:20.735057 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:20.735061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:20.735993 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:20.736005 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:20.736912 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:20.736920 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:20.737061 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:20.737935 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:20.737954 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:20.741356 (+ 3402us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":293,"threads_started":1}
W20260620 02:53:20.741847 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:20.732328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50627 (local address 127.0.97.62:45373)
0620 02:53:20.732616 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:20.732623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:20.733041 (+ 418us) server_negotiation.cc:407] Connection header received
0620 02:53:20.733099 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:20.733102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:20.733196 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:20.733282 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:20.734129 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:20.734864 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:20.736165 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:20.736688 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:20.736808 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:20.741553 (+ 4745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:50627: BlockingRecv error: recv got EOF from 127.0.97.1:50627 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 02:53:21 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:21 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:21.225852 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:21.217557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.2:54091)
0620 02:53:21.218083 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:21.218103 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:21.218246 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:21.218707 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:21.218712 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:21.219109 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:21.219261 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:21.219269 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:21.220383 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:21.220388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:21.221337 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:21.221349 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:21.222131 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:21.222137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:21.222253 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:21.222873 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:21.222892 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:21.225632 (+ 2740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":254,"threads_started":1}
W20260620 02:53:21.226132 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:21.217666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54091 (local address 127.0.97.62:45373)
0620 02:53:21.217880 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:21.217889 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:21.218236 (+ 347us) server_negotiation.cc:407] Connection header received
0620 02:53:21.218405 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:21.218411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:21.218542 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:21.218655 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:21.219399 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:21.220211 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:21.221493 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:21.221961 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:21.222064 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:21.225848 (+ 3784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:54091: BlockingRecv error: recv got EOF from 127.0.97.2:54091 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 20 02:53:21 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:21 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:21.709105 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:21.699202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.3:51457)
0620 02:53:21.699769 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:21.699787 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:21.699903 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:21.700320 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:21.700329 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:21.700957 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:21.701199 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:21.701211 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:21.702876 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:21.702882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:21.704243 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:21.704256 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:21.705064 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:21.705071 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:21.705262 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:21.705835 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:21.705853 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:21.708880 (+ 3027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":285,"threads_started":1}
W20260620 02:53:21.709348 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:21.699269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51457 (local address 127.0.97.62:45373)
0620 02:53:21.699471 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:21.699479 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:21.699868 (+ 389us) server_negotiation.cc:407] Connection header received
0620 02:53:21.700021 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:21.700029 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:21.700127 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:21.700255 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:21.701385 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:21.702733 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:21.704391 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:21.704865 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:21.705015 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:21.709081 (+ 4066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:51457: BlockingRecv error: recv got EOF from 127.0.97.3:51457 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 02:53:21 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:21 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:21.751897 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:21.742865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.1:57529)
0620 02:53:21.743432 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:21.743452 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:21.743597 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:21.744144 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:21.744148 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:21.744519 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:21.744661 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:21.744669 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:21.745726 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:21.745731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:21.746707 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:21.746719 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:21.747509 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:21.747515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:21.747669 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:21.748404 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:21.748422 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:21.751563 (+ 3141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":535,"threads_started":1}
W20260620 02:53:21.752168 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:21.743059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57529 (local address 127.0.97.62:45373)
0620 02:53:21.743368 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:21.743374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:21.743571 (+ 197us) server_negotiation.cc:407] Connection header received
0620 02:53:21.743902 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:21.743908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:21.743995 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:21.744090 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:21.744807 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:21.745579 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:21.746869 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:21.747333 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:21.747435 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:21.751815 (+ 4380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57529: BlockingRecv error: recv got EOF from 127.0.97.1:57529 (error 108)
Metrics: {"server-negotiator.queue_time_us":183}
Jun 20 02:53:22 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:22 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:22.236141 9692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:22.226914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.2:36115)
0620 02:53:22.227472 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:22.227503 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:22.227676 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:22.228122 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:22.228127 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:22.228592 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:22.228824 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:22.228835 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:22.229996 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:22.230002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:22.230923 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:22.230939 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:22.231659 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:22.231664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:22.231842 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:22.232574 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:22.232595 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:22.235871 (+ 3276us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":320,"threads_started":1}
W20260620 02:53:22.236403 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:22.227172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36115 (local address 127.0.97.62:45373)
0620 02:53:22.227432 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:22.227440 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:22.227632 (+ 192us) server_negotiation.cc:407] Connection header received
0620 02:53:22.227860 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:22.227866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:22.227955 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:22.228095 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:22.228997 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:22.229845 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:22.231077 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:22.231513 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:22.231649 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:22.236107 (+ 4458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:36115: BlockingRecv error: recv got EOF from 127.0.97.2:36115 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 20 02:53:22 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:22 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:22.718912 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:22.710275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.3:50677)
0620 02:53:22.710770 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:22.710787 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:22.710905 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:22.711243 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:22.711247 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:22.711611 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:22.711798 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:22.711806 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:22.712843 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:22.712847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:22.713953 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:22.713965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:22.714800 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:22.714806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:22.714999 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:22.715746 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:22.715764 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:22.718671 (+ 2907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":289,"threads_started":1}
W20260620 02:53:22.719121 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:22.710286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50677 (local address 127.0.97.62:45373)
0620 02:53:22.710583 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:22.710591 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:22.710889 (+ 298us) server_negotiation.cc:407] Connection header received
0620 02:53:22.711009 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:22.711013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:22.711072 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:22.711193 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:22.711966 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:22.712718 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:22.714144 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:22.714638 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:22.714780 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:22.718888 (+ 4108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:50677: BlockingRecv error: recv got EOF from 127.0.97.3:50677 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Jun 20 02:53:22 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:22 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:22.761606 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:22.753226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.1:50331)
0620 02:53:22.753683 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:22.753700 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:22.753806 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:22.754165 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:22.754168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:22.754620 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:22.754785 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:22.754793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:22.755953 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:22.755969 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:22.757002 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:22.757013 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:22.757719 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:22.757724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:22.757898 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:22.758586 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:22.758603 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:22.761254 (+ 2651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":265,"threads_started":1}
W20260620 02:53:22.761675 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:22.753267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50331 (local address 127.0.97.62:45373)
0620 02:53:22.753522 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:22.753530 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:22.753782 (+ 252us) server_negotiation.cc:407] Connection header received
0620 02:53:22.753909 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:22.753915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:22.754004 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:22.754094 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:22.754949 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:22.755814 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:22.757145 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:22.757576 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:22.757706 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:22.761436 (+ 3730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:50331: BlockingRecv error: recv got EOF from 127.0.97.1:50331 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 02:53:23 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 02:53:23 dist-test-slave-8p8c krb5kdc[8648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 02:53:23.246441 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:23.237521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45373 (local address 127.0.97.2:53517)
0620 02:53:23.238141 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:23.238163 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:23.238309 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:23.238719 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:23.238723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:23.239277 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:23.239464 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:23.239472 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:23.240495 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:23.240500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:23.241428 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:23.241439 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:23.242158 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:23.242163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:23.242335 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:23.243017 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:23.243034 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:23.246133 (+ 3099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45373: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":392,"threads_started":1}
W20260620 02:53:23.246677 9386 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:23.237675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53517 (local address 127.0.97.62:45373)
0620 02:53:23.237860 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:23.237869 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:23.238239 (+ 370us) server_negotiation.cc:407] Connection header received
0620 02:53:23.238498 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:23.238503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:23.238583 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:23.238688 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:23.239605 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:23.240353 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:23.241564 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:23.242004 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:23.242148 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:23.246400 (+ 4252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:53517: BlockingRecv error: recv got EOF from 127.0.97.2:53517 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 02:53:23.544620 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 9237
I20260620 02:53:23.559738 9697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:23.545903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54186 (local address 127.0.97.1:45371)
0620 02:53:23.546503 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:23.546510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:23.546541 (+ 31us) server_negotiation.cc:407] Connection header received
0620 02:53:23.546607 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:23.546614 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:23.546873 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:23.547144 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:23.549096 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:23.550211 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:23.551663 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:23.552535 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:23.552830 (+ 295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:23.555852 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:23.555862 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:23.555877 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:53:23.555923 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:23.559156 (+ 3233us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 02:53:23.559475 (+ 319us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54186: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":270,"threads_started":1}
W20260620 02:53:23.559870 9697 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54186: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 02:53:23.560002 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 9237: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:45371: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 02:53:23.560258 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 9237
I20260620 02:53:23.577131 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 9388
I20260620 02:53:23.590030 9698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:23.578532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48134 (local address 127.0.97.2:45001)
0620 02:53:23.579141 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:23.579150 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:23.579182 (+ 32us) server_negotiation.cc:407] Connection header received
0620 02:53:23.579247 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:23.579254 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:23.579489 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:23.579672 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:23.580938 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:23.582103 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:23.583397 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:23.584013 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:23.584126 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:23.586765 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:23.586774 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:23.586788 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:53:23.586826 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:23.589423 (+ 2597us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 02:53:23.589688 (+ 265us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48134: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":299,"threads_started":1}
W20260620 02:53:23.590133 9698 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48134: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 02:53:23.590188 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 9388: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:45001: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 02:53:23.590353 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 9388
I20260620 02:53:23.608371 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 9519
I20260620 02:53:23.620952 9699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:23.609414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56744 (local address 127.0.97.3:41687)
0620 02:53:23.609991 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:23.609999 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:23.610028 (+ 29us) server_negotiation.cc:407] Connection header received
0620 02:53:23.610114 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:23.610121 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:23.610341 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:23.610598 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:23.611937 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:23.613021 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:23.614309 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:23.614986 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:23.615096 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:23.617605 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:23.617614 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:23.617628 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:53:23.617674 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:23.620411 (+ 2737us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 02:53:23.620674 (+ 263us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56744: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":274,"threads_started":1}
W20260620 02:53:23.621063 9699 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56744: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 02:53:23.621170 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 9519: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:41687: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 02:53:23.621342 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 9519
I20260620 02:53:23.637804 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 9168
I20260620 02:53:23.639221 9230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:53:23.755226 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 9168
2026-06-20T02:53:23Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21293 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T02:53:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:53:23Z Disabled control of system clock
I20260620 02:53:23.829409 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35535
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35535
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 02:53:24.165855 9704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:53:24.166194 9704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:53:24.166278 9704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:53:24.174086 9704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:53:24.174220 9704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:53:24.174257 9704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:53:24.174307 9704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:53:24.174351 9704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:53:24.185073 9704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35535
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:53:24.186641 9704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:53:24.188602 9704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:53:24.201694 9709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:24.201766 9712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:24.201733 9710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:53:24.202594 9704 server_base.cc:1034] running on GCE node
I20260620 02:53:24.203415 9704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:53:24.204866 9704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:53:24.206126 9704 hybrid_clock.cc:648] HybridClock initialized: now 1781924004206068 us; error 88 us; skew 500 ppm
I20260620 02:53:24.206558 9704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:53:24.208997 9704 webserver.cc:460] Webserver started at http://127.0.97.62:42497/ using document root <none> and password file <none>
I20260620 02:53:24.209843 9704 fs_manager.cc:359] Metadata directory not provided
I20260620 02:53:24.209982 9704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:53:24.210323 9704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:53:24.213382 9704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "c687077f02594b9b9edd1116c99aceaf"
format_stamp: "Formatted at 2026-06-20 02:53:24 on dist-test-slave-8p8c"
server_key: "d3e090d376268829474e1bbf796eb11a"
server_key_iv: "53827c6ffbc60424744e5222789583fd"
server_key_version: "encryptionkey@0"
I20260620 02:53:24.214257 9704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "c687077f02594b9b9edd1116c99aceaf"
format_stamp: "Formatted at 2026-06-20 02:53:24 on dist-test-slave-8p8c"
server_key: "d3e090d376268829474e1bbf796eb11a"
server_key_iv: "53827c6ffbc60424744e5222789583fd"
server_key_version: "encryptionkey@0"
I20260620 02:53:24.219758 9704 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 02:53:24.223551 9718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:24.225085 9704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:53:24.225335 9704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "c687077f02594b9b9edd1116c99aceaf"
format_stamp: "Formatted at 2026-06-20 02:53:24 on dist-test-slave-8p8c"
server_key: "d3e090d376268829474e1bbf796eb11a"
server_key_iv: "53827c6ffbc60424744e5222789583fd"
server_key_version: "encryptionkey@0"
I20260620 02:53:24.225570 9704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:53:24.258926 9704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:53:24.259908 9704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:53:24.260277 9704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:53:24.280596 9704 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35535
I20260620 02:53:24.280656 9769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35535 every 8 connection(s)
I20260620 02:53:24.282157 9704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:53:24.286196 9770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:53:24.291308 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 9704
I20260620 02:53:24.291558 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:53:24.292250 388 external_mini_cluster.cc:1428] Setting key f9cabaf95c0ca2036d64319553449b30
I20260620 02:53:24.298486 9770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c687077f02594b9b9edd1116c99aceaf: Bootstrap starting.
I20260620 02:53:24.302878 9770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c687077f02594b9b9edd1116c99aceaf: Neither blocks nor log segments found. Creating new log.
I20260620 02:53:24.304175 9770 log.cc:824] T 00000000000000000000000000000000 P c687077f02594b9b9edd1116c99aceaf: Log is configured to *not* fsync() on all Append() calls
I20260620 02:53:24.308137 9770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c687077f02594b9b9edd1116c99aceaf: No bootstrap required, opened a new log
I20260620 02:53:24.309795 9773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:24.294680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41606 (local address 127.0.97.62:35535)
0620 02:53:24.295869 (+ 1189us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:24.295885 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:24.295936 (+ 51us) server_negotiation.cc:407] Connection header received
0620 02:53:24.298199 (+ 2263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:24.298229 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:24.298866 (+ 637us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:24.299313 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:24.299817 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:24.302128 (+ 2311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:24.303719 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:24.304949 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:24.305118 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:24.306701 (+ 1583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:24.306714 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:24.306736 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:53:24.307524 (+ 788us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:24.307641 (+ 117us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:53:24.307671 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:53:24.307805 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:24.308037 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 02:53:24.308385 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":385,"threads_started":1}
I20260620 02:53:24.317023 9770 raft_consensus.cc:348] T 00000000000000000000000000000000 P c687077f02594b9b9edd1116c99aceaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c687077f02594b9b9edd1116c99aceaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35535 } }
I20260620 02:53:24.317453 9770 raft_consensus.cc:374] T 00000000000000000000000000000000 P c687077f02594b9b9edd1116c99aceaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:53:24.317595 9770 raft_consensus.cc:729] T 00000000000000000000000000000000 P c687077f02594b9b9edd1116c99aceaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c687077f02594b9b9edd1116c99aceaf, State: Initialized, Role: FOLLOWER
I20260620 02:53:24.318286 9770 consensus_queue.cc:260] T 00000000000000000000000000000000 P c687077f02594b9b9edd1116c99aceaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c687077f02594b9b9edd1116c99aceaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35535 } }
I20260620 02:53:24.318665 9770 raft_consensus.cc:388] T 00000000000000000000000000000000 P c687077f02594b9b9edd1116c99aceaf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:53:24.318825 9770 raft_consensus.cc:482] T 00000000000000000000000000000000 P c687077f02594b9b9edd1116c99aceaf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:53:24.319028 9770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c687077f02594b9b9edd1116c99aceaf [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:53:24.321527 9770 raft_consensus.cc:504] T 00000000000000000000000000000000 P c687077f02594b9b9edd1116c99aceaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c687077f02594b9b9edd1116c99aceaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35535 } }
I20260620 02:53:24.322057 9770 leader_election.cc:302] T 00000000000000000000000000000000 P c687077f02594b9b9edd1116c99aceaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c687077f02594b9b9edd1116c99aceaf; no voters:
I20260620 02:53:24.322718 9770 leader_election.cc:288] T 00000000000000000000000000000000 P c687077f02594b9b9edd1116c99aceaf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:53:24.322889 9775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c687077f02594b9b9edd1116c99aceaf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:53:24.323714 9775 raft_consensus.cc:686] T 00000000000000000000000000000000 P c687077f02594b9b9edd1116c99aceaf [term 1 LEADER]: Becoming Leader. State: Replica: c687077f02594b9b9edd1116c99aceaf, State: Running, Role: LEADER
I20260620 02:53:24.324390 9775 consensus_queue.cc:237] T 00000000000000000000000000000000 P c687077f02594b9b9edd1116c99aceaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c687077f02594b9b9edd1116c99aceaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35535 } }
I20260620 02:53:24.325040 9770 sys_catalog.cc:564] T 00000000000000000000000000000000 P c687077f02594b9b9edd1116c99aceaf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:53:24.327750 9777 sys_catalog.cc:455] T 00000000000000000000000000000000 P c687077f02594b9b9edd1116c99aceaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader c687077f02594b9b9edd1116c99aceaf. Latest consensus state: current_term: 1 leader_uuid: "c687077f02594b9b9edd1116c99aceaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c687077f02594b9b9edd1116c99aceaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35535 } } }
I20260620 02:53:24.328174 9777 sys_catalog.cc:458] T 00000000000000000000000000000000 P c687077f02594b9b9edd1116c99aceaf [sys.catalog]: This master's current role is: LEADER
I20260620 02:53:24.327993 9776 sys_catalog.cc:455] T 00000000000000000000000000000000 P c687077f02594b9b9edd1116c99aceaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c687077f02594b9b9edd1116c99aceaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c687077f02594b9b9edd1116c99aceaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35535 } } }
I20260620 02:53:24.328322 9776 sys_catalog.cc:458] T 00000000000000000000000000000000 P c687077f02594b9b9edd1116c99aceaf [sys.catalog]: This master's current role is: LEADER
I20260620 02:53:24.331784 9784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:53:24.336948 9784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:53:24.345304 9784 catalog_manager.cc:1347] Generated new cluster ID: f30539b3706a49748ff4d61293fdd0bd
I20260620 02:53:24.345479 9784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:53:24.357410 9784 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:53:24.358892 9784 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:53:24.371130 9784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c687077f02594b9b9edd1116c99aceaf: Generated new TSK 0
I20260620 02:53:24.372036 9784 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:53:24.384294 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35535
--builtin_ntp_servers=127.0.97.20:33057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 02:53:24.724807 9794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:53:24.725103 9794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:53:24.725243 9794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:53:24.733029 9794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:53:24.733126 9794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:53:24.733227 9794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:53:24.742645 9794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:53:24.743894 9794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:53:24.745628 9794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:53:24.759532 9799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:24.759616 9800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:24.759645 9802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:53:24.760370 9794 server_base.cc:1034] running on GCE node
I20260620 02:53:24.761029 9794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:53:24.762099 9794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:53:24.763365 9794 hybrid_clock.cc:648] HybridClock initialized: now 1781924004763243 us; error 134 us; skew 500 ppm
I20260620 02:53:24.763777 9794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:53:24.769889 9794 webserver.cc:460] Webserver started at http://127.0.97.1:33839/ using document root <none> and password file <none>
I20260620 02:53:24.770714 9794 fs_manager.cc:359] Metadata directory not provided
I20260620 02:53:24.770851 9794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:53:24.771184 9794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:53:24.774072 9794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/data/instance:
uuid: "537557baf6424a39baf47f038b65d3f9"
format_stamp: "Formatted at 2026-06-20 02:53:24 on dist-test-slave-8p8c"
server_key: "98dc48268011a35a7b90046444980a75"
server_key_iv: "ea99307ab5eeec1e7005002057543020"
server_key_version: "encryptionkey@0"
I20260620 02:53:24.774909 9794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/wal/instance:
uuid: "537557baf6424a39baf47f038b65d3f9"
format_stamp: "Formatted at 2026-06-20 02:53:24 on dist-test-slave-8p8c"
server_key: "98dc48268011a35a7b90046444980a75"
server_key_iv: "ea99307ab5eeec1e7005002057543020"
server_key_version: "encryptionkey@0"
I20260620 02:53:24.780061 9794 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 02:53:24.783758 9808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:24.785282 9794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:53:24.785486 9794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/wal
uuid: "537557baf6424a39baf47f038b65d3f9"
format_stamp: "Formatted at 2026-06-20 02:53:24 on dist-test-slave-8p8c"
server_key: "98dc48268011a35a7b90046444980a75"
server_key_iv: "ea99307ab5eeec1e7005002057543020"
server_key_version: "encryptionkey@0"
I20260620 02:53:24.785733 9794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:53:24.807763 9794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:53:24.808802 9794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:53:24.809149 9794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:53:24.810174 9794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:53:24.811969 9794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:53:24.812115 9794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:24.812260 9794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:53:24.812356 9794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:24.845710 9794 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35083
I20260620 02:53:24.845763 9920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35083 every 8 connection(s)
I20260620 02:53:24.847092 9794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:53:24.852650 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 9794
I20260620 02:53:24.852864 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:53:24.853513 388 external_mini_cluster.cc:1428] Setting key b2f6620caa3b897051ba2e4e6eb2205f
I20260620 02:53:24.856515 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35535
--builtin_ntp_servers=127.0.97.20:33057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 02:53:24.860819 9923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:24.849833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35181 (local address 127.0.97.62:35535)
0620 02:53:24.850281 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:24.850286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:24.851664 (+ 1378us) server_negotiation.cc:407] Connection header received
0620 02:53:24.853490 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:24.853497 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:24.853630 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:24.853759 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:24.854971 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:24.856105 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:24.857695 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:24.858452 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:24.858589 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:24.859846 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:24.859855 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:24.859863 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:53:24.859887 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:24.859950 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:53:24.859967 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:53:24.860059 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:24.860288 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 02:53:24.860393 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":259,"threads_started":1}
I20260620 02:53:24.861593 9924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:24.850541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35535 (local address 127.0.97.1:35181)
0620 02:53:24.851497 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:24.851532 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:24.853233 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:24.853981 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:24.853989 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:24.854074 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:24.854774 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:24.854797 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:24.856274 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:24.856278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:24.857541 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:24.857553 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:24.858643 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:24.858648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:24.858845 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:24.859549 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:53:24.859552 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:24.859589 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:53:24.859666 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:53:24.860062 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:53:24.860068 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:53:24.860073 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 02:53:24.860247 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 02:53:24.860526 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":283,"threads_started":1}
I20260620 02:53:24.863117 9921 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35535
I20260620 02:53:24.863405 9921 heartbeater.cc:464] Registering TS with master...
I20260620 02:53:24.864179 9921 heartbeater.cc:511] Master 127.0.97.62:35535 requested a full tablet report, sending...
I20260620 02:53:24.866109 9735 ts_manager.cc:194] Registered new tserver with Master: 537557baf6424a39baf47f038b65d3f9 (127.0.97.1:35083)
I20260620 02:53:24.867633 9735 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:35181
W20260620 02:53:25.184661 9925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:53:25.184950 9925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:53:25.185089 9925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:53:25.192552 9925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:53:25.192663 9925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:53:25.192832 9925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:53:25.202033 9925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:53:25.203323 9925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:53:25.205098 9925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:53:25.218600 9930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:25.218551 9933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:25.218647 9931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:53:25.224118 9925 server_base.cc:1034] running on GCE node
I20260620 02:53:25.224856 9925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:53:25.225936 9925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:53:25.227165 9925 hybrid_clock.cc:648] HybridClock initialized: now 1781924005227096 us; error 57 us; skew 500 ppm
I20260620 02:53:25.227536 9925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:53:25.230147 9925 webserver.cc:460] Webserver started at http://127.0.97.2:38055/ using document root <none> and password file <none>
I20260620 02:53:25.230978 9925 fs_manager.cc:359] Metadata directory not provided
I20260620 02:53:25.231067 9925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:53:25.231418 9925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:53:25.234336 9925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/data/instance:
uuid: "40fbb4813399434dbae1658018d49fa9"
format_stamp: "Formatted at 2026-06-20 02:53:25 on dist-test-slave-8p8c"
server_key: "e783c52a7d89fd22a27880dbb55a785a"
server_key_iv: "57ad475a866392683ef7862affdde379"
server_key_version: "encryptionkey@0"
I20260620 02:53:25.235227 9925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/wal/instance:
uuid: "40fbb4813399434dbae1658018d49fa9"
format_stamp: "Formatted at 2026-06-20 02:53:25 on dist-test-slave-8p8c"
server_key: "e783c52a7d89fd22a27880dbb55a785a"
server_key_iv: "57ad475a866392683ef7862affdde379"
server_key_version: "encryptionkey@0"
I20260620 02:53:25.240509 9925 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 02:53:25.244184 9939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:25.245728 9925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:53:25.245924 9925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/wal
uuid: "40fbb4813399434dbae1658018d49fa9"
format_stamp: "Formatted at 2026-06-20 02:53:25 on dist-test-slave-8p8c"
server_key: "e783c52a7d89fd22a27880dbb55a785a"
server_key_iv: "57ad475a866392683ef7862affdde379"
server_key_version: "encryptionkey@0"
I20260620 02:53:25.246150 9925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:53:25.261089 9925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:53:25.262023 9925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:53:25.262320 9925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:53:25.263411 9925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:53:25.265030 9925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:53:25.265117 9925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:25.265280 9925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:53:25.265362 9925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:25.300241 9925 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37553
I20260620 02:53:25.300292 10051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37553 every 8 connection(s)
I20260620 02:53:25.301620 9925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:53:25.310348 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 9925
I20260620 02:53:25.310571 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:53:25.311167 388 external_mini_cluster.cc:1428] Setting key cda9ef0057a3d7088852aaf19f705270
I20260620 02:53:25.314481 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35535
--builtin_ntp_servers=127.0.97.20:33057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 02:53:25.315650 9923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:25.304608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57947 (local address 127.0.97.62:35535)
0620 02:53:25.304918 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:25.304925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:25.306200 (+ 1275us) server_negotiation.cc:407] Connection header received
0620 02:53:25.307969 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:25.307973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:25.308042 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:25.308139 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:25.309321 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:25.310064 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:25.312365 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:25.312963 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:25.313124 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:25.314777 (+ 1653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:25.314786 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:25.314794 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:53:25.314818 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:25.314859 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:53:25.314877 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:53:25.314940 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:25.315319 (+ 379us) server_negotiation.cc:299] Negotiation successful
0620 02:53:25.315419 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
I20260620 02:53:25.317200 10054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:25.305117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35535 (local address 127.0.97.2:57947)
0620 02:53:25.306054 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:25.306090 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:25.307718 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:25.308381 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:25.308390 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:25.308480 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:25.309095 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:25.309118 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:25.310225 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:25.310230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:25.312212 (+ 1982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:25.312224 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:25.313213 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:25.313218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:25.313514 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:25.314333 (+ 819us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:53:25.314337 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:25.314424 (+ 87us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:53:25.314536 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:53:25.315023 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:53:25.315051 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:53:25.315060 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 02:53:25.315250 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 02:53:25.315726 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":285,"threads_started":1}
I20260620 02:53:25.318797 10052 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35535
I20260620 02:53:25.319128 10052 heartbeater.cc:464] Registering TS with master...
I20260620 02:53:25.320057 10052 heartbeater.cc:511] Master 127.0.97.62:35535 requested a full tablet report, sending...
I20260620 02:53:25.321580 9735 ts_manager.cc:194] Registered new tserver with Master: 40fbb4813399434dbae1658018d49fa9 (127.0.97.2:37553)
I20260620 02:53:25.322866 9735 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:57947
W20260620 02:53:25.645591 10055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:53:25.645875 10055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:53:25.645958 10055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:53:25.653319 10055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:53:25.653430 10055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:53:25.653535 10055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:53:25.662647 10055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:53:25.663916 10055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:53:25.665609 10055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:53:25.679134 10061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:25.679077 10063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:25.679193 10060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:53:25.681599 10055 server_base.cc:1034] running on GCE node
I20260620 02:53:25.682543 10055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:53:25.683651 10055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:53:25.684904 10055 hybrid_clock.cc:648] HybridClock initialized: now 1781924005684846 us; error 58 us; skew 500 ppm
I20260620 02:53:25.685297 10055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:53:25.687903 10055 webserver.cc:460] Webserver started at http://127.0.97.3:39037/ using document root <none> and password file <none>
I20260620 02:53:25.688664 10055 fs_manager.cc:359] Metadata directory not provided
I20260620 02:53:25.688797 10055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:53:25.689131 10055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:53:25.692180 10055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/data/instance:
uuid: "c249a641ff6849c18fe3769fed5798f2"
format_stamp: "Formatted at 2026-06-20 02:53:25 on dist-test-slave-8p8c"
server_key: "bce375ada11ce73c1735942e30d31c25"
server_key_iv: "3ef27a3e5d987fdc0d88d7f8c9719cb4"
server_key_version: "encryptionkey@0"
I20260620 02:53:25.693017 10055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c249a641ff6849c18fe3769fed5798f2"
format_stamp: "Formatted at 2026-06-20 02:53:25 on dist-test-slave-8p8c"
server_key: "bce375ada11ce73c1735942e30d31c25"
server_key_iv: "3ef27a3e5d987fdc0d88d7f8c9719cb4"
server_key_version: "encryptionkey@0"
I20260620 02:53:25.699219 10055 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 02:53:25.703217 10069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:25.704759 10055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:53:25.704967 10055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/wal
uuid: "c249a641ff6849c18fe3769fed5798f2"
format_stamp: "Formatted at 2026-06-20 02:53:25 on dist-test-slave-8p8c"
server_key: "bce375ada11ce73c1735942e30d31c25"
server_key_iv: "3ef27a3e5d987fdc0d88d7f8c9719cb4"
server_key_version: "encryptionkey@0"
I20260620 02:53:25.705200 10055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:53:25.724848 10055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:53:25.725852 10055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:53:25.726145 10055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:53:25.727164 10055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:53:25.728709 10055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:53:25.728834 10055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:25.728992 10055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:53:25.729079 10055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:25.764122 10055 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45293
I20260620 02:53:25.764205 10181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45293 every 8 connection(s)
I20260620 02:53:25.765759 10055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:53:25.769934 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 10055
I20260620 02:53:25.770099 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:53:25.770612 388 external_mini_cluster.cc:1428] Setting key 96c95f878b36cd163d1fbe041af9360f
I20260620 02:53:25.778642 9923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:25.768328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60315 (local address 127.0.97.62:35535)
0620 02:53:25.768542 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:25.768547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:25.770301 (+ 1754us) server_negotiation.cc:407] Connection header received
0620 02:53:25.772404 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:25.772409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:25.772492 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:25.772575 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:25.773658 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:25.774418 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:25.775932 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:25.776351 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:25.776481 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:25.777781 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:25.777786 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:25.777790 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:53:25.777806 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:25.777836 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:53:25.777847 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:53:25.777941 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:25.778195 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 02:53:25.778285 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 02:53:25.779636 10184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:25.768890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35535 (local address 127.0.97.3:60315)
0620 02:53:25.770103 (+ 1213us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:25.770166 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:25.772110 (+ 1944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:25.772774 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:25.772782 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:25.772869 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:25.773497 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:25.773513 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:25.774580 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:25.774585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:25.775788 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:25.775799 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:25.776561 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:25.776569 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:25.776793 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:25.777471 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:53:25.777474 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:25.777511 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:53:25.777589 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:53:25.777938 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:53:25.777950 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:53:25.777959 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 02:53:25.778188 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 02:53:25.778579 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":281,"threads_started":1}
I20260620 02:53:25.780970 10182 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35535
I20260620 02:53:25.781252 10182 heartbeater.cc:464] Registering TS with master...
I20260620 02:53:25.781970 10182 heartbeater.cc:511] Master 127.0.97.62:35535 requested a full tablet report, sending...
I20260620 02:53:25.783406 9735 ts_manager.cc:194] Registered new tserver with Master: c249a641ff6849c18fe3769fed5798f2 (127.0.97.3:45293)
I20260620 02:53:25.784258 9735 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:60315
I20260620 02:53:25.786258 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 02:53:25.792869 9923 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:25.791159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41616 (local address 127.0.97.62:35535)
0620 02:53:25.791378 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:25.791388 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:25.791791 (+ 403us) server_negotiation.cc:407] Connection header received
0620 02:53:25.791948 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:25.791952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:25.792025 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:25.792129 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:25.792690 (+ 561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41616: BlockingRecv error: recv got EOF from 127.0.0.1:41616 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:53:25.793419 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 9794
I20260620 02:53:25.801869 10191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:25.794623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53244 (local address 127.0.97.1:35083)
0620 02:53:25.795155 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:25.795164 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:25.795207 (+ 43us) server_negotiation.cc:407] Connection header received
0620 02:53:25.795263 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:25.795269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:25.795489 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:25.795617 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:25.796025 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:25.797039 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:25.798339 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:25.799149 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:25.799256 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:25.800437 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:25.800445 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:25.800458 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:53:25.801129 (+ 671us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:25.801179 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:53:25.801199 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:53:25.801293 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:25.801467 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 02:53:25.801584 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":313,"threads_started":1}
I20260620 02:53:25.803241 9916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:53:25.888388 9921 heartbeater.cc:503] Master 127.0.97.62:35535 was elected leader, sending a full tablet report...
I20260620 02:53:25.889655 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 9794
W20260620 02:53:25.905978 9721 connection.cc:535] server connection from 127.0.97.1:35181 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 02:53:25.906279 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 9925
I20260620 02:53:25.914706 10193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:25.907161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40516 (local address 127.0.97.2:37553)
0620 02:53:25.907706 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:25.907714 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:25.907739 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:53:25.907804 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:25.907810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:25.908028 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:25.908222 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:25.908615 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:25.909677 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:25.911058 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:25.911671 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:25.911816 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:25.912945 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:25.912954 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:25.912968 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:53:25.913775 (+ 807us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:25.913829 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:53:25.913858 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:53:25.914030 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:25.914192 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 02:53:25.914346 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":301,"threads_started":1}
I20260620 02:53:25.916126 10047 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:53:26.002981 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 9925
I20260620 02:53:26.019671 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 10055
I20260620 02:53:26.028162 10195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:26.020763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55818 (local address 127.0.97.3:45293)
0620 02:53:26.021476 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:26.021483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:26.021515 (+ 32us) server_negotiation.cc:407] Connection header received
0620 02:53:26.021580 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:26.021586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:26.021808 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:26.022002 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:26.022490 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:26.023514 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:26.024747 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:26.025385 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:26.025523 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:26.026721 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:26.026731 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:26.026745 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:53:26.027326 (+ 581us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:26.027372 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:53:26.027391 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:53:26.027516 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:26.027662 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 02:53:26.027802 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":401,"threads_started":1}
I20260620 02:53:26.029497 10177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:53:26.111079 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 10055
I20260620 02:53:26.126669 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 9704
I20260620 02:53:26.127892 9765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:53:26.229346 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 9704
2026-06-20T02:53:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2469 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T02:53:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:53:26Z Disabled control of system clock
I20260620 02:53:26.301832 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42055
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 02:53:26.625758 10201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:53:26.626127 10201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:53:26.626291 10201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:53:26.633558 10201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:53:26.633716 10201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:53:26.633814 10201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:53:26.633927 10201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:53:26.634014 10201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:53:26.643209 10201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42055
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:53:26.644594 10201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:53:26.646458 10201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:53:26.658159 10207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:26.658115 10206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:26.658197 10209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:53:26.659468 10201 server_base.cc:1034] running on GCE node
I20260620 02:53:26.660374 10201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:53:26.661914 10201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:53:26.663151 10201 hybrid_clock.cc:648] HybridClock initialized: now 1781924006663105 us; error 38 us; skew 500 ppm
I20260620 02:53:26.663558 10201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:53:26.666231 10201 webserver.cc:460] Webserver started at http://127.0.97.62:37623/ using document root <none> and password file <none>
I20260620 02:53:26.666993 10201 fs_manager.cc:359] Metadata directory not provided
I20260620 02:53:26.667081 10201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:53:26.667411 10201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:53:26.670327 10201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "ac39299793084b1a9e75a24f2f448483"
format_stamp: "Formatted at 2026-06-20 02:53:26 on dist-test-slave-8p8c"
server_key: "c76cc1f1e3723774975ac924a2e72ba1"
server_key_iv: "e9d6edf8901f4019b227930cbd34680c"
server_key_version: "encryptionkey@0"
I20260620 02:53:26.671136 10201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "ac39299793084b1a9e75a24f2f448483"
format_stamp: "Formatted at 2026-06-20 02:53:26 on dist-test-slave-8p8c"
server_key: "c76cc1f1e3723774975ac924a2e72ba1"
server_key_iv: "e9d6edf8901f4019b227930cbd34680c"
server_key_version: "encryptionkey@0"
I20260620 02:53:26.676151 10201 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:53:26.679486 10215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:26.680819 10201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:53:26.681080 10201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "ac39299793084b1a9e75a24f2f448483"
format_stamp: "Formatted at 2026-06-20 02:53:26 on dist-test-slave-8p8c"
server_key: "c76cc1f1e3723774975ac924a2e72ba1"
server_key_iv: "e9d6edf8901f4019b227930cbd34680c"
server_key_version: "encryptionkey@0"
I20260620 02:53:26.681284 10201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:53:26.687546 10201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:53:26.688356 10201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:53:26.688683 10201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:53:26.708586 10201 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42055
I20260620 02:53:26.708609 10266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42055 every 8 connection(s)
I20260620 02:53:26.710067 10201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:53:26.713006 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 10201
I20260620 02:53:26.713238 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:53:26.714000 388 external_mini_cluster.cc:1428] Setting key ed46ebdbc9581d5ebd70e30e88cd018b
I20260620 02:53:26.714432 10267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:53:26.722963 10270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:26.716216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54058 (local address 127.0.97.62:42055)
0620 02:53:26.717326 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:26.717338 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:26.717386 (+ 48us) server_negotiation.cc:407] Connection header received
0620 02:53:26.719260 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:26.719285 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:26.719742 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:26.720136 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:26.721244 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:26.721250 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:26.721263 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:53:26.721278 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:26.721335 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:53:26.721350 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:53:26.721538 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:26.721654 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 02:53:26.721977 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":420,"threads_started":1}
I20260620 02:53:26.727484 10267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac39299793084b1a9e75a24f2f448483: Bootstrap starting.
I20260620 02:53:26.730284 10267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac39299793084b1a9e75a24f2f448483: Neither blocks nor log segments found. Creating new log.
I20260620 02:53:26.731244 10267 log.cc:824] T 00000000000000000000000000000000 P ac39299793084b1a9e75a24f2f448483: Log is configured to *not* fsync() on all Append() calls
I20260620 02:53:26.734045 10267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac39299793084b1a9e75a24f2f448483: No bootstrap required, opened a new log
I20260620 02:53:26.741484 10267 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac39299793084b1a9e75a24f2f448483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac39299793084b1a9e75a24f2f448483" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42055 } }
I20260620 02:53:26.741879 10267 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac39299793084b1a9e75a24f2f448483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:53:26.742002 10267 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac39299793084b1a9e75a24f2f448483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac39299793084b1a9e75a24f2f448483, State: Initialized, Role: FOLLOWER
I20260620 02:53:26.742661 10267 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac39299793084b1a9e75a24f2f448483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac39299793084b1a9e75a24f2f448483" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42055 } }
I20260620 02:53:26.742862 10267 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac39299793084b1a9e75a24f2f448483 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:53:26.742949 10267 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac39299793084b1a9e75a24f2f448483 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:53:26.743101 10267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac39299793084b1a9e75a24f2f448483 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:53:26.745611 10267 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac39299793084b1a9e75a24f2f448483 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac39299793084b1a9e75a24f2f448483" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42055 } }
I20260620 02:53:26.746191 10267 leader_election.cc:302] T 00000000000000000000000000000000 P ac39299793084b1a9e75a24f2f448483 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac39299793084b1a9e75a24f2f448483; no voters:
I20260620 02:53:26.746872 10267 leader_election.cc:288] T 00000000000000000000000000000000 P ac39299793084b1a9e75a24f2f448483 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:53:26.746997 10272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac39299793084b1a9e75a24f2f448483 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:53:26.747750 10272 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac39299793084b1a9e75a24f2f448483 [term 1 LEADER]: Becoming Leader. State: Replica: ac39299793084b1a9e75a24f2f448483, State: Running, Role: LEADER
I20260620 02:53:26.748289 10272 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac39299793084b1a9e75a24f2f448483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac39299793084b1a9e75a24f2f448483" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42055 } }
I20260620 02:53:26.749118 10267 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac39299793084b1a9e75a24f2f448483 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:53:26.751325 10274 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac39299793084b1a9e75a24f2f448483 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac39299793084b1a9e75a24f2f448483. Latest consensus state: current_term: 1 leader_uuid: "ac39299793084b1a9e75a24f2f448483" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac39299793084b1a9e75a24f2f448483" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42055 } } }
I20260620 02:53:26.751690 10274 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac39299793084b1a9e75a24f2f448483 [sys.catalog]: This master's current role is: LEADER
I20260620 02:53:26.751442 10273 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac39299793084b1a9e75a24f2f448483 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac39299793084b1a9e75a24f2f448483" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac39299793084b1a9e75a24f2f448483" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42055 } } }
I20260620 02:53:26.751907 10273 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac39299793084b1a9e75a24f2f448483 [sys.catalog]: This master's current role is: LEADER
I20260620 02:53:26.755805 10279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:53:26.761432 10279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:53:26.772033 10279 catalog_manager.cc:1347] Generated new cluster ID: 3d6afb527b61422ab65b17ff85d3320e
I20260620 02:53:26.772166 10279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:53:26.795251 10279 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:53:26.795760 10279 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:53:26.803645 10279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac39299793084b1a9e75a24f2f448483: Generated new TSK 0
I20260620 02:53:26.804768 10279 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:53:26.821694 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42055
--builtin_ntp_servers=127.0.97.20:39305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 02:53:27.152755 10291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:53:27.153053 10291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:53:27.153151 10291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:53:27.161269 10291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:53:27.161381 10291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:53:27.161502 10291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:53:27.171321 10291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:53:27.172695 10291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:53:27.174535 10291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:53:27.189109 10297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:27.189208 10296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:27.189227 10299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:53:27.189982 10291 server_base.cc:1034] running on GCE node
I20260620 02:53:27.190658 10291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:53:27.191665 10291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:53:27.192916 10291 hybrid_clock.cc:648] HybridClock initialized: now 1781924007192860 us; error 48 us; skew 500 ppm
I20260620 02:53:27.193260 10291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:53:27.195696 10291 webserver.cc:460] Webserver started at http://127.0.97.1:46139/ using document root <none> and password file <none>
I20260620 02:53:27.196412 10291 fs_manager.cc:359] Metadata directory not provided
I20260620 02:53:27.196539 10291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:53:27.196871 10291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:53:27.199916 10291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/data/instance:
uuid: "5b12c611388e4b90ad36869ea8d78802"
format_stamp: "Formatted at 2026-06-20 02:53:27 on dist-test-slave-8p8c"
server_key: "5513b34d993ce54fd4b85c24e9a6fdf8"
server_key_iv: "88f1ad985fc334921d6093bf8370fc9f"
server_key_version: "encryptionkey@0"
I20260620 02:53:27.200695 10291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5b12c611388e4b90ad36869ea8d78802"
format_stamp: "Formatted at 2026-06-20 02:53:27 on dist-test-slave-8p8c"
server_key: "5513b34d993ce54fd4b85c24e9a6fdf8"
server_key_iv: "88f1ad985fc334921d6093bf8370fc9f"
server_key_version: "encryptionkey@0"
I20260620 02:53:27.205888 10291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 02:53:27.209439 10305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:27.211138 10291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:53:27.211333 10291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/wal
uuid: "5b12c611388e4b90ad36869ea8d78802"
format_stamp: "Formatted at 2026-06-20 02:53:27 on dist-test-slave-8p8c"
server_key: "5513b34d993ce54fd4b85c24e9a6fdf8"
server_key_iv: "88f1ad985fc334921d6093bf8370fc9f"
server_key_version: "encryptionkey@0"
I20260620 02:53:27.211643 10291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:53:27.217460 10291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:53:27.218310 10291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:53:27.218634 10291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:53:27.219633 10291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:53:27.221061 10291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:53:27.221180 10291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:27.221311 10291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:53:27.221396 10291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:27.255722 10291 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37811
I20260620 02:53:27.255849 10417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37811 every 8 connection(s)
I20260620 02:53:27.257117 10291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:53:27.264038 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 10291
I20260620 02:53:27.264259 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:53:27.265053 388 external_mini_cluster.cc:1428] Setting key 7f399967b316cf65fe92760ec38cd7d2
I20260620 02:53:27.265851 10420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:27.259806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38283 (local address 127.0.97.62:42055)
0620 02:53:27.260254 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:27.260258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:27.261475 (+ 1217us) server_negotiation.cc:407] Connection header received
0620 02:53:27.263185 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:27.263190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:27.263244 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:27.263365 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:27.265030 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:27.265035 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:27.265038 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:53:27.265056 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:27.265126 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:53:27.265144 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:53:27.265296 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:27.265512 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 02:53:27.265575 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":247,"threads_started":1}
I20260620 02:53:27.266888 10421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:27.260422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42055 (local address 127.0.97.1:38283)
0620 02:53:27.261324 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:27.261359 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:27.262910 (+ 1551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:27.263684 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:27.263699 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:27.263829 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:27.264642 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:53:27.264646 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:27.264702 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:53:27.264819 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:53:27.265258 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:53:27.265267 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:53:27.265276 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 02:53:27.265556 (+ 280us) client_negotiation.cc:240] Negotiation successful
0620 02:53:27.265798 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":265,"threads_started":1}
I20260620 02:53:27.267926 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42055
--builtin_ntp_servers=127.0.97.20:39305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 02:53:27.268409 10418 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42055
I20260620 02:53:27.268728 10418 heartbeater.cc:464] Registering TS with master...
I20260620 02:53:27.269336 10418 heartbeater.cc:511] Master 127.0.97.62:42055 requested a full tablet report, sending...
I20260620 02:53:27.271486 10232 ts_manager.cc:194] Registered new tserver with Master: 5b12c611388e4b90ad36869ea8d78802 (127.0.97.1:37811)
W20260620 02:53:27.588600 10422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:53:27.588901 10422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:53:27.589067 10422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:53:27.596199 10422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:53:27.596329 10422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:53:27.596517 10422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:53:27.605541 10422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:53:27.606850 10422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:53:27.608668 10422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:53:27.621575 10428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:27.621570 10427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:27.621635 10430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:53:27.622598 10422 server_base.cc:1034] running on GCE node
I20260620 02:53:27.623160 10422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:53:27.623980 10422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:53:27.625213 10422 hybrid_clock.cc:648] HybridClock initialized: now 1781924007625152 us; error 53 us; skew 500 ppm
I20260620 02:53:27.625540 10422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:53:27.627871 10422 webserver.cc:460] Webserver started at http://127.0.97.2:37597/ using document root <none> and password file <none>
I20260620 02:53:27.628544 10422 fs_manager.cc:359] Metadata directory not provided
I20260620 02:53:27.628670 10422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:53:27.629038 10422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:53:27.631877 10422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/data/instance:
uuid: "a66cbb27cbdc4d6e89da8e087664fff0"
format_stamp: "Formatted at 2026-06-20 02:53:27 on dist-test-slave-8p8c"
server_key: "ae641970696f7158a924579bed017d24"
server_key_iv: "547cb991ac86d0727dcd1f5f68515e3c"
server_key_version: "encryptionkey@0"
I20260620 02:53:27.632653 10422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a66cbb27cbdc4d6e89da8e087664fff0"
format_stamp: "Formatted at 2026-06-20 02:53:27 on dist-test-slave-8p8c"
server_key: "ae641970696f7158a924579bed017d24"
server_key_iv: "547cb991ac86d0727dcd1f5f68515e3c"
server_key_version: "encryptionkey@0"
I20260620 02:53:27.637625 10422 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:53:27.641211 10436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:27.642594 10422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:53:27.642760 10422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/wal
uuid: "a66cbb27cbdc4d6e89da8e087664fff0"
format_stamp: "Formatted at 2026-06-20 02:53:27 on dist-test-slave-8p8c"
server_key: "ae641970696f7158a924579bed017d24"
server_key_iv: "547cb991ac86d0727dcd1f5f68515e3c"
server_key_version: "encryptionkey@0"
I20260620 02:53:27.643023 10422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:53:27.648409 10422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:53:27.649217 10422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:53:27.649492 10422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:53:27.650502 10422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:53:27.651839 10422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:53:27.651921 10422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:27.652004 10422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:53:27.652058 10422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:27.684923 10422 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42599
I20260620 02:53:27.684983 10548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42599 every 8 connection(s)
I20260620 02:53:27.686451 10422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:53:27.690318 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 10422
I20260620 02:53:27.690523 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:53:27.691004 388 external_mini_cluster.cc:1428] Setting key 844e335a43455b72830e7db1c72b570e
I20260620 02:53:27.693778 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42055
--builtin_ntp_servers=127.0.97.20:39305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 02:53:27.695140 10420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:27.689075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57345 (local address 127.0.97.62:42055)
0620 02:53:27.689294 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:27.689300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:27.690884 (+ 1584us) server_negotiation.cc:407] Connection header received
0620 02:53:27.692849 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:27.692854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:27.692915 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:27.693041 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:27.694409 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:27.694414 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:27.694419 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:53:27.694442 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:27.694472 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:53:27.694483 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:53:27.694604 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:27.694806 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 02:53:27.694870 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 02:53:27.696077 10551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:27.689619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42055 (local address 127.0.97.2:57345)
0620 02:53:27.690718 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:27.690765 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:27.692575 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:27.693269 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:27.693277 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:27.693371 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:27.694092 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:53:27.694095 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:27.694136 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:53:27.694223 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:53:27.694579 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:53:27.694589 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:53:27.694596 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 02:53:27.694880 (+ 284us) client_negotiation.cc:240] Negotiation successful
0620 02:53:27.695145 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":309,"threads_started":1}
I20260620 02:53:27.697445 10549 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42055
I20260620 02:53:27.697739 10549 heartbeater.cc:464] Registering TS with master...
I20260620 02:53:27.698318 10549 heartbeater.cc:511] Master 127.0.97.62:42055 requested a full tablet report, sending...
I20260620 02:53:27.699827 10232 ts_manager.cc:194] Registered new tserver with Master: a66cbb27cbdc4d6e89da8e087664fff0 (127.0.97.2:42599)
W20260620 02:53:28.029162 10552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:53:28.029512 10552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:53:28.029685 10552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:53:28.036912 10552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:53:28.037055 10552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:53:28.037240 10552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:53:28.047083 10552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:53:28.048403 10552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:53:28.050442 10552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:53:28.064136 10560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:28.064092 10557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:28.064185 10558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:53:28.069432 10552 server_base.cc:1034] running on GCE node
I20260620 02:53:28.070142 10552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:53:28.071271 10552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:53:28.072530 10552 hybrid_clock.cc:648] HybridClock initialized: now 1781924008072456 us; error 46 us; skew 500 ppm
I20260620 02:53:28.072873 10552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:53:28.075280 10552 webserver.cc:460] Webserver started at http://127.0.97.3:34029/ using document root <none> and password file <none>
I20260620 02:53:28.075980 10552 fs_manager.cc:359] Metadata directory not provided
I20260620 02:53:28.076108 10552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:53:28.076426 10552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:53:28.079430 10552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/data/instance:
uuid: "f2d9e94565f14d57bb2b1923ba26a3da"
format_stamp: "Formatted at 2026-06-20 02:53:28 on dist-test-slave-8p8c"
server_key: "6e57af758c7f9770b0f23371857cf55d"
server_key_iv: "f27c18215b0b4e69064b184e96197e9b"
server_key_version: "encryptionkey@0"
I20260620 02:53:28.080217 10552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f2d9e94565f14d57bb2b1923ba26a3da"
format_stamp: "Formatted at 2026-06-20 02:53:28 on dist-test-slave-8p8c"
server_key: "6e57af758c7f9770b0f23371857cf55d"
server_key_iv: "f27c18215b0b4e69064b184e96197e9b"
server_key_version: "encryptionkey@0"
I20260620 02:53:28.085297 10552 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:53:28.088980 10566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:28.090260 10552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:53:28.090443 10552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/wal
uuid: "f2d9e94565f14d57bb2b1923ba26a3da"
format_stamp: "Formatted at 2026-06-20 02:53:28 on dist-test-slave-8p8c"
server_key: "6e57af758c7f9770b0f23371857cf55d"
server_key_iv: "f27c18215b0b4e69064b184e96197e9b"
server_key_version: "encryptionkey@0"
I20260620 02:53:28.090714 10552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:53:28.096190 10552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:53:28.096942 10552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:53:28.097203 10552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:53:28.098085 10552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:53:28.099469 10552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:53:28.099597 10552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:28.099730 10552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:53:28.099815 10552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:28.132975 10552 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35239
I20260620 02:53:28.133018 10678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35239 every 8 connection(s)
I20260620 02:53:28.134265 10552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:53:28.137485 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 10552
I20260620 02:53:28.137674 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781923914418765-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:53:28.138217 388 external_mini_cluster.cc:1428] Setting key 447d855fa655bd5a9ad8195baf56df77
I20260620 02:53:28.142891 10420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:28.136925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41245 (local address 127.0.97.62:42055)
0620 02:53:28.137126 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:28.137130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:28.138675 (+ 1545us) server_negotiation.cc:407] Connection header received
0620 02:53:28.140567 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:28.140572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:28.140639 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:28.140759 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:28.142127 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:28.142133 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:28.142137 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:53:28.142155 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:28.142196 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:53:28.142207 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:53:28.142323 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:28.142566 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 02:53:28.142629 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 02:53:28.143792 10681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:28.137468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42055 (local address 127.0.97.3:41245)
0620 02:53:28.138516 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:28.138565 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:28.140317 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:28.141004 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:28.141016 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:28.141146 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:28.141846 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:53:28.141849 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:28.141885 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:53:28.141964 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:53:28.142298 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:53:28.142308 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:53:28.142316 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 02:53:28.142643 (+ 327us) client_negotiation.cc:240] Negotiation successful
0620 02:53:28.142885 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":314,"threads_started":1}
I20260620 02:53:28.145205 10679 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42055
I20260620 02:53:28.145519 10679 heartbeater.cc:464] Registering TS with master...
I20260620 02:53:28.146188 10679 heartbeater.cc:511] Master 127.0.97.62:42055 requested a full tablet report, sending...
I20260620 02:53:28.147660 10232 ts_manager.cc:194] Registered new tserver with Master: f2d9e94565f14d57bb2b1923ba26a3da (127.0.97.3:35239)
I20260620 02:53:28.153288 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 02:53:28.159206 10420 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:53:28.157684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54068 (local address 127.0.97.62:42055)
0620 02:53:28.157838 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:28.157842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:28.158310 (+ 468us) server_negotiation.cc:407] Connection header received
0620 02:53:28.158500 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:28.158503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:28.158583 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:28.158698 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:28.159013 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54068: BlockingRecv error: recv got EOF from 127.0.0.1:54068 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 02:53:28.159925 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 10291
I20260620 02:53:28.163220 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:28.160794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58498 (local address 127.0.97.1:37811)
0620 02:53:28.161287 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:28.161293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:28.161319 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:53:28.161372 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:28.161378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:28.161591 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:28.161710 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:28.162623 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:28.162629 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:28.162641 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:53:28.162660 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:28.162705 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:53:28.162719 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:53:28.162818 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:28.162916 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 02:53:28.162992 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":239,"threads_started":1}
I20260620 02:53:28.164523 10413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:53:28.245131 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 10291
I20260620 02:53:28.262197 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 10422
I20260620 02:53:28.266116 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:28.263155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39166 (local address 127.0.97.2:42599)
0620 02:53:28.263678 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:28.263685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:28.263713 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:53:28.263776 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:28.263782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:28.264007 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:28.264138 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:28.265362 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:28.265370 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:28.265383 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:53:28.265399 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:28.265453 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:53:28.265470 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:53:28.265614 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:28.265751 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 02:53:28.265841 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":242,"threads_started":1}
I20260620 02:53:28.267416 10544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:53:28.349376 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 10422
I20260620 02:53:28.367231 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 10552
I20260620 02:53:28.371594 10692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:28.368359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33112 (local address 127.0.97.3:35239)
0620 02:53:28.369093 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:28.369100 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:28.369157 (+ 57us) server_negotiation.cc:407] Connection header received
0620 02:53:28.369251 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:28.369266 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:28.369501 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:28.369672 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:28.370854 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:28.370861 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:28.370873 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:53:28.370889 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:28.370926 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:53:28.370939 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:53:28.371130 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:28.371205 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 02:53:28.371304 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":419,"threads_started":1}
I20260620 02:53:28.372920 10674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:53:28.459416 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 10552
I20260620 02:53:28.475418 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 10201
I20260620 02:53:28.476614 10262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:53:28.577863 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 10201
2026-06-20T02:53:28Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2348 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:53:28 dist-test-slave-8p8c krb5kdc[10697](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:53:28 dist-test-slave-8p8c krb5kdc[10697](info): set up 2 sockets
Jun 20 02:53:28 dist-test-slave-8p8c krb5kdc[10697](info): commencing operation
krb5kdc: starting...
W20260620 02:53:29.679381 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:53:29 dist-test-slave-8p8c krb5kdc[10697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:53:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:53:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:53:29.931244 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36143
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36143
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:53:30.270099 10710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:53:30.270498 10710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:53:30.270660 10710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:53:30.277926 10710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:53:30.278147 10710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:53:30.278404 10710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:53:30.278537 10710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:53:30.278631 10710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:53:30.288590 10710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:53:30.290026 10710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:53:30.291435 10710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:53:30.304734 10718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:30.304798 10716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:30.304827 10715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:53:30.305510 10710 server_base.cc:1034] running on GCE node
I20260620 02:53:30.306319 10710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:53:30.307941 10710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:53:30.309166 10710 hybrid_clock.cc:648] HybridClock initialized: now 1781924010309091 us; error 62 us; skew 500 ppm
I20260620 02:53:30.309590 10710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:53:30 dist-test-slave-8p8c krb5kdc[10697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:53:30.317022 10710 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:53:30.318687 10710 webserver.cc:460] Webserver started at http://127.0.97.62:45479/ using document root <none> and password file <none>
I20260620 02:53:30.319425 10710 fs_manager.cc:359] Metadata directory not provided
I20260620 02:53:30.319552 10710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:53:30.319876 10710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:53:30.323381 10710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "5818e441e02d48fb9758513ef52a409c"
format_stamp: "Formatted at 2026-06-20 02:53:30 on dist-test-slave-8p8c"
server_key: "fbc2584db69024123347f4b0c687f237"
server_key_iv: "b184699e108e764763d40dc961ea4a1e"
server_key_version: "encryptionkey@0"
I20260620 02:53:30.324164 10710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "5818e441e02d48fb9758513ef52a409c"
format_stamp: "Formatted at 2026-06-20 02:53:30 on dist-test-slave-8p8c"
server_key: "fbc2584db69024123347f4b0c687f237"
server_key_iv: "b184699e108e764763d40dc961ea4a1e"
server_key_version: "encryptionkey@0"
I20260620 02:53:30.329396 10710 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 02:53:30.333030 10725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:30.334620 10710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:53:30.334803 10710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "5818e441e02d48fb9758513ef52a409c"
format_stamp: "Formatted at 2026-06-20 02:53:30 on dist-test-slave-8p8c"
server_key: "fbc2584db69024123347f4b0c687f237"
server_key_iv: "b184699e108e764763d40dc961ea4a1e"
server_key_version: "encryptionkey@0"
I20260620 02:53:30.335038 10710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:53:30.352046 10710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:53:30.356861 10710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:53:30.357175 10710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:53:30.375892 10710 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36143
I20260620 02:53:30.375931 10776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36143 every 8 connection(s)
I20260620 02:53:30.377282 10710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:53:30.381690 10777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:53:30.385294 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 10710
I20260620 02:53:30.385550 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:53:30.386286 388 external_mini_cluster.cc:1428] Setting key d1e872679cba0e38196dde9aecadd81d
I20260620 02:53:30.395175 10777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5818e441e02d48fb9758513ef52a409c: Bootstrap starting.
I20260620 02:53:30.398447 10777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5818e441e02d48fb9758513ef52a409c: Neither blocks nor log segments found. Creating new log.
I20260620 02:53:30.399482 10777 log.cc:824] T 00000000000000000000000000000000 P 5818e441e02d48fb9758513ef52a409c: Log is configured to *not* fsync() on all Append() calls
Jun 20 02:53:30 dist-test-slave-8p8c krb5kdc[10697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:53:30.402446 10777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5818e441e02d48fb9758513ef52a409c: No bootstrap required, opened a new log
I20260620 02:53:30.409796 10780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:30.388410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49338 (local address 127.0.97.62:36143)
0620 02:53:30.389383 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:30.389395 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:30.389439 (+ 44us) server_negotiation.cc:407] Connection header received
0620 02:53:30.391336 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:30.391364 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:30.391834 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:30.392286 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:30.393750 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:30.395220 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:30.396693 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:30.397400 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:30.397607 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:30.401709 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:30.401716 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:30.401731 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:53:30.401765 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:30.405063 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:30.406001 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:30.406008 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:30.406014 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:30.406187 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:30.406698 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:30.406702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:30.406704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:30.407515 (+ 811us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:53:30.407731 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:30.408275 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 02:53:30.408728 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":500,"threads_started":1}
I20260620 02:53:30.411623 10777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5818e441e02d48fb9758513ef52a409c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5818e441e02d48fb9758513ef52a409c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36143 } }
I20260620 02:53:30.412115 10777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5818e441e02d48fb9758513ef52a409c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:53:30.412230 10777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5818e441e02d48fb9758513ef52a409c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5818e441e02d48fb9758513ef52a409c, State: Initialized, Role: FOLLOWER
I20260620 02:53:30.412870 10777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5818e441e02d48fb9758513ef52a409c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5818e441e02d48fb9758513ef52a409c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36143 } }
I20260620 02:53:30.413125 10777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5818e441e02d48fb9758513ef52a409c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:53:30.413285 10777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5818e441e02d48fb9758513ef52a409c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:53:30.413445 10777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5818e441e02d48fb9758513ef52a409c [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:53:30.415906 10777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5818e441e02d48fb9758513ef52a409c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5818e441e02d48fb9758513ef52a409c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36143 } }
I20260620 02:53:30.416424 10777 leader_election.cc:302] T 00000000000000000000000000000000 P 5818e441e02d48fb9758513ef52a409c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5818e441e02d48fb9758513ef52a409c; no voters:
I20260620 02:53:30.417069 10777 leader_election.cc:288] T 00000000000000000000000000000000 P 5818e441e02d48fb9758513ef52a409c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:53:30.417251 10782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5818e441e02d48fb9758513ef52a409c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:53:30.418160 10782 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5818e441e02d48fb9758513ef52a409c [term 1 LEADER]: Becoming Leader. State: Replica: 5818e441e02d48fb9758513ef52a409c, State: Running, Role: LEADER
I20260620 02:53:30.418695 10782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5818e441e02d48fb9758513ef52a409c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5818e441e02d48fb9758513ef52a409c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36143 } }
I20260620 02:53:30.419268 10777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5818e441e02d48fb9758513ef52a409c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:53:30.421376 10783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5818e441e02d48fb9758513ef52a409c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5818e441e02d48fb9758513ef52a409c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5818e441e02d48fb9758513ef52a409c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36143 } } }
I20260620 02:53:30.422015 10783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5818e441e02d48fb9758513ef52a409c [sys.catalog]: This master's current role is: LEADER
I20260620 02:53:30.421939 10784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5818e441e02d48fb9758513ef52a409c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5818e441e02d48fb9758513ef52a409c. Latest consensus state: current_term: 1 leader_uuid: "5818e441e02d48fb9758513ef52a409c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5818e441e02d48fb9758513ef52a409c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36143 } } }
I20260620 02:53:30.422237 10784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5818e441e02d48fb9758513ef52a409c [sys.catalog]: This master's current role is: LEADER
I20260620 02:53:30.424482 10787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:53:30.430028 10787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:53:30.440577 10787 catalog_manager.cc:1347] Generated new cluster ID: bee5e8ec36724a028e6a2709bb4d6a38
I20260620 02:53:30.440702 10787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:53:30.449607 10787 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:53:30.451478 10787 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:53:30.467250 10787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5818e441e02d48fb9758513ef52a409c: Generated new TSK 0
I20260620 02:53:30.468192 10787 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:53:30.539304 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36143
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:53:30.877331 10803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:53:30.877708 10803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:53:30.877898 10803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:53:30.885417 10803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:53:30.885603 10803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:53:30.885780 10803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:53:30.895929 10803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:53:30.897334 10803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:53:30.898742 10803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:53:30.912932 10811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:30.912973 10808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:30.913009 10809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:53:30.913899 10803 server_base.cc:1034] running on GCE node
I20260620 02:53:30.914664 10803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:53:30.915692 10803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:53:30.916918 10803 hybrid_clock.cc:648] HybridClock initialized: now 1781924010916826 us; error 80 us; skew 500 ppm
I20260620 02:53:30.917475 10803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:53:30 dist-test-slave-8p8c krb5kdc[10697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:53:30.922160 10803 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:53:30.923804 10803 webserver.cc:460] Webserver started at http://127.0.97.1:43331/ using document root <none> and password file <none>
I20260620 02:53:30.924605 10803 fs_manager.cc:359] Metadata directory not provided
I20260620 02:53:30.924741 10803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:53:30.925156 10803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:53:30.928682 10803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-0/data/instance:
uuid: "c229720ea53a414dab151c7651284d9e"
format_stamp: "Formatted at 2026-06-20 02:53:30 on dist-test-slave-8p8c"
server_key: "4f0aa409d0ecb28c7e6abb43fed882b2"
server_key_iv: "fdb59bacb5680584358db0ba3c96acca"
server_key_version: "encryptionkey@0"
I20260620 02:53:30.929557 10803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c229720ea53a414dab151c7651284d9e"
format_stamp: "Formatted at 2026-06-20 02:53:30 on dist-test-slave-8p8c"
server_key: "4f0aa409d0ecb28c7e6abb43fed882b2"
server_key_iv: "fdb59bacb5680584358db0ba3c96acca"
server_key_version: "encryptionkey@0"
I20260620 02:53:30.935230 10803 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 02:53:30.939181 10818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:30.941181 10803 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:53:30.941386 10803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-0/wal
uuid: "c229720ea53a414dab151c7651284d9e"
format_stamp: "Formatted at 2026-06-20 02:53:30 on dist-test-slave-8p8c"
server_key: "4f0aa409d0ecb28c7e6abb43fed882b2"
server_key_iv: "fdb59bacb5680584358db0ba3c96acca"
server_key_version: "encryptionkey@0"
I20260620 02:53:30.941612 10803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:53:30.955036 10803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:53:30.959758 10803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:53:30.960072 10803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:53:30.962070 10803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:53:30.963606 10803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:53:30.963711 10803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:30.963841 10803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:53:30.963917 10803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:31.003871 10803 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34697
I20260620 02:53:31.003909 10930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34697 every 8 connection(s)
I20260620 02:53:31.005743 10803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:53:31.015583 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 10803
I20260620 02:53:31.015830 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:53:31.016661 388 external_mini_cluster.cc:1428] Setting key 65208e23fac698a654409169d4f2a898
Jun 20 02:53:31 dist-test-slave-8p8c krb5kdc[10697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:53:31.028700 10933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:31.008595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35863 (local address 127.0.97.62:36143)
0620 02:53:31.009058 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:31.009062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:31.010098 (+ 1036us) server_negotiation.cc:407] Connection header received
0620 02:53:31.012235 (+ 2137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:31.012272 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:31.012384 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:31.012505 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:31.014562 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:31.015316 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:31.017677 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:31.018224 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:31.018400 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:31.022528 (+ 4128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:31.022533 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:31.022537 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:53:31.022569 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:31.025511 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:31.026310 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:31.026316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:31.026320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:31.026507 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:31.026965 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:31.026970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:31.026974 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:31.027244 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:53:31.027407 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:31.028117 (+ 710us) server_negotiation.cc:299] Negotiation successful
0620 02:53:31.028330 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":359,"thread_start_us":232,"threads_started":1}
I20260620 02:53:31.029970 10934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:31.009140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36143 (local address 127.0.97.1:35863)
0620 02:53:31.009876 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:31.009919 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:31.011715 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:31.013043 (+ 1328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:31.013052 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:31.013687 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:31.014313 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:31.014331 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:31.015530 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:31.015537 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:31.017525 (+ 1988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:31.017537 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:31.018587 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:31.018601 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:31.018859 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:31.019598 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:31.019622 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:31.022253 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:53:31.025674 (+ 3421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:53:31.025683 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:53:31.025702 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:53:31.026174 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:53:31.026619 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:53:31.026624 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:53:31.026628 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:53:31.026828 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:53:31.027374 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:53:31.027384 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:53:31.027753 (+ 369us) client_negotiation.cc:769] Sending connection context
0620 02:53:31.028014 (+ 261us) client_negotiation.cc:240] Negotiation successful
0620 02:53:31.028580 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":298,"threads_started":1}
I20260620 02:53:31.031922 10931 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36143
I20260620 02:53:31.032299 10931 heartbeater.cc:464] Registering TS with master...
I20260620 02:53:31.033102 10931 heartbeater.cc:511] Master 127.0.97.62:36143 requested a full tablet report, sending...
I20260620 02:53:31.035652 10742 ts_manager.cc:194] Registered new tserver with Master: c229720ea53a414dab151c7651284d9e (127.0.97.1:34697)
I20260620 02:53:31.037145 10742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35863
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:53:31.079270 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36143
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:53:31.425287 10937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:53:31.425590 10937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:53:31.425745 10937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:53:31.433274 10937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:53:31.433394 10937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:53:31.433630 10937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:53:31.444682 10937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:53:31.446223 10937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:53:31.447659 10937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:53:31.460748 10942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:31.460812 10943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:31.461146 10945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:53:31.461687 10937 server_base.cc:1034] running on GCE node
I20260620 02:53:31.462555 10937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:53:31.463691 10937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:53:31.464941 10937 hybrid_clock.cc:648] HybridClock initialized: now 1781924011464897 us; error 32 us; skew 500 ppm
I20260620 02:53:31.465291 10937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:53:31 dist-test-slave-8p8c krb5kdc[10697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:53:31.468659 10937 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:53:31.470517 10937 webserver.cc:460] Webserver started at http://127.0.97.2:39379/ using document root <none> and password file <none>
I20260620 02:53:31.471241 10937 fs_manager.cc:359] Metadata directory not provided
I20260620 02:53:31.471390 10937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:53:31.471707 10937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:53:31.474769 10937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-1/data/instance:
uuid: "2e6b52e9ceed4b6db4cccd54f99342af"
format_stamp: "Formatted at 2026-06-20 02:53:31 on dist-test-slave-8p8c"
server_key: "4b4f6c82eaa06c3b8e6ab82249bbe5df"
server_key_iv: "39aef1e5ac699bc441c3963473dfd089"
server_key_version: "encryptionkey@0"
I20260620 02:53:31.475529 10937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2e6b52e9ceed4b6db4cccd54f99342af"
format_stamp: "Formatted at 2026-06-20 02:53:31 on dist-test-slave-8p8c"
server_key: "4b4f6c82eaa06c3b8e6ab82249bbe5df"
server_key_iv: "39aef1e5ac699bc441c3963473dfd089"
server_key_version: "encryptionkey@0"
I20260620 02:53:31.480908 10937 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:53:31.484506 10952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:31.486274 10937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:53:31.486488 10937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-1/wal
uuid: "2e6b52e9ceed4b6db4cccd54f99342af"
format_stamp: "Formatted at 2026-06-20 02:53:31 on dist-test-slave-8p8c"
server_key: "4b4f6c82eaa06c3b8e6ab82249bbe5df"
server_key_iv: "39aef1e5ac699bc441c3963473dfd089"
server_key_version: "encryptionkey@0"
I20260620 02:53:31.486878 10937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:53:31.499724 10937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:53:31.504074 10937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:53:31.504377 10937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:53:31.505368 10937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:53:31.506843 10937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:53:31.506944 10937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:31.507091 10937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:53:31.507164 10937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:31.545390 10937 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37285
I20260620 02:53:31.545557 11064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37285 every 8 connection(s)
I20260620 02:53:31.546927 10937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:53:31.554157 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 10937
I20260620 02:53:31.554374 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:53:31.554839 388 external_mini_cluster.cc:1428] Setting key 616546a8c08a4611a44092086391cff5
Jun 20 02:53:31 dist-test-slave-8p8c krb5kdc[10697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:53:31.568337 11067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:31.549709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47249 (local address 127.0.97.62:36143)
0620 02:53:31.550182 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:31.550188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:31.551258 (+ 1070us) server_negotiation.cc:407] Connection header received
0620 02:53:31.553045 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:31.553050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:31.553119 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:31.553222 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:31.555054 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:31.555951 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:31.557509 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:31.557906 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:31.558042 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:31.561980 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:31.561986 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:31.561989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:53:31.562030 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:31.564861 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:31.565719 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:31.565726 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:31.565736 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:31.565863 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:31.566406 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:31.566411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:31.566417 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:31.566757 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:53:31.566923 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:31.567676 (+ 753us) server_negotiation.cc:299] Negotiation successful
0620 02:53:31.567923 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":216,"threads_started":1}
I20260620 02:53:31.569430 11068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:31.550305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36143 (local address 127.0.97.2:47249)
0620 02:53:31.551092 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:31.551132 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:31.552721 (+ 1589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:31.553518 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:31.553527 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:31.554070 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:31.554824 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:31.554843 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:31.556098 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:31.556104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:31.557353 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:31.557364 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:31.558131 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:31.558138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:31.558427 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:31.559215 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:31.559238 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:31.561729 (+ 2491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:53:31.565048 (+ 3319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:53:31.565058 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:53:31.565077 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:53:31.565569 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:53:31.566020 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:53:31.566027 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:53:31.566031 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:53:31.566248 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:53:31.566895 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:53:31.566903 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:53:31.567276 (+ 373us) client_negotiation.cc:769] Sending connection context
0620 02:53:31.567576 (+ 300us) client_negotiation.cc:240] Negotiation successful
0620 02:53:31.568197 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":313,"threads_started":1}
I20260620 02:53:31.571094 11065 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36143
I20260620 02:53:31.571455 11065 heartbeater.cc:464] Registering TS with master...
I20260620 02:53:31.572414 11065 heartbeater.cc:511] Master 127.0.97.62:36143 requested a full tablet report, sending...
I20260620 02:53:31.574329 10742 ts_manager.cc:194] Registered new tserver with Master: 2e6b52e9ceed4b6db4cccd54f99342af (127.0.97.2:37285)
I20260620 02:53:31.575742 10742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47249
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:53:31.610052 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36143
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:53:31.951864 11071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:53:31.952172 11071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:53:31.952304 11071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:53:31.959930 11071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:53:31.960062 11071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:53:31.960258 11071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:53:31.972307 11071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:53:31.974117 11071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:53:31.975639 11071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:53:31.990897 11076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:31.990970 11077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:53:31.990854 11079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:53:31.991227 11071 server_base.cc:1034] running on GCE node
I20260620 02:53:31.992228 11071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:53:31.993254 11071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:53:31.994516 11071 hybrid_clock.cc:648] HybridClock initialized: now 1781924011994472 us; error 35 us; skew 500 ppm
I20260620 02:53:31.994900 11071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:53:31 dist-test-slave-8p8c krb5kdc[10697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:53:31.998643 11071 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:53:32.000350 11071 webserver.cc:460] Webserver started at http://127.0.97.3:44897/ using document root <none> and password file <none>
I20260620 02:53:32.001163 11071 fs_manager.cc:359] Metadata directory not provided
I20260620 02:53:32.001291 11071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:53:32.001631 11071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:53:32.004932 11071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-2/data/instance:
uuid: "9879444722a2436a879c65466130b7ec"
format_stamp: "Formatted at 2026-06-20 02:53:31 on dist-test-slave-8p8c"
server_key: "02c65a159d19dbce557e6b8421af571e"
server_key_iv: "11aab13c6bfb0eb652246d5618ddc5e8"
server_key_version: "encryptionkey@0"
I20260620 02:53:32.005823 11071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9879444722a2436a879c65466130b7ec"
format_stamp: "Formatted at 2026-06-20 02:53:31 on dist-test-slave-8p8c"
server_key: "02c65a159d19dbce557e6b8421af571e"
server_key_iv: "11aab13c6bfb0eb652246d5618ddc5e8"
server_key_version: "encryptionkey@0"
I20260620 02:53:32.011205 11071 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 02:53:32.015010 11086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:32.016444 11071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 02:53:32.016615 11071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-2/wal
uuid: "9879444722a2436a879c65466130b7ec"
format_stamp: "Formatted at 2026-06-20 02:53:31 on dist-test-slave-8p8c"
server_key: "02c65a159d19dbce557e6b8421af571e"
server_key_iv: "11aab13c6bfb0eb652246d5618ddc5e8"
server_key_version: "encryptionkey@0"
I20260620 02:53:32.016801 11071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:53:32.040689 10931 heartbeater.cc:503] Master 127.0.97.62:36143 was elected leader, sending a full tablet report...
I20260620 02:53:32.048949 11071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:53:32.053985 11071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:53:32.054315 11071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:53:32.055474 11071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:53:32.057186 11071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:53:32.057302 11071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:32.057467 11071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:53:32.057551 11071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:53:32.095918 11071 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46505
I20260620 02:53:32.096019 11198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46505 every 8 connection(s)
I20260620 02:53:32.097241 11071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:53:32.103482 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 11071
I20260620 02:53:32.103696 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:53:32.104113 388 external_mini_cluster.cc:1428] Setting key 28ec703fb733f1e47f5441ae0b857d34
Jun 20 02:53:32 dist-test-slave-8p8c krb5kdc[10697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:53:32.121641 11201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:32.100074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50013 (local address 127.0.97.62:36143)
0620 02:53:32.100499 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:32.100504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:32.101678 (+ 1174us) server_negotiation.cc:407] Connection header received
0620 02:53:32.103980 (+ 2302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:32.103985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:32.104082 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:32.104205 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:32.105963 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.107339 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.109289 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.109721 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.109886 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:32.114298 (+ 4412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:32.114303 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:32.114307 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:53:32.114347 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:32.117789 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:32.118793 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:32.118802 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:32.118808 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:32.118974 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:32.119560 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:32.119566 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:32.119579 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:32.120117 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:53:32.120287 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:32.121110 (+ 823us) server_negotiation.cc:299] Negotiation successful
0620 02:53:32.121374 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":229,"threads_started":1}
I20260620 02:53:32.122805 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:32.100551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36143 (local address 127.0.97.3:50013)
0620 02:53:32.101417 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:32.101458 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:32.103654 (+ 2196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:32.104457 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:32.104465 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:32.105133 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:53:32.105798 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:32.105817 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.107492 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.107502 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:32.109124 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:32.109138 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.110222 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.110229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:32.110706 (+ 477us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:32.111513 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:53:32.111537 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:53:32.114051 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:53:32.117951 (+ 3900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:53:32.117962 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:53:32.117993 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:53:32.118542 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:53:32.119080 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:53:32.119087 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:53:32.119090 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:53:32.119375 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:53:32.120271 (+ 896us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:53:32.120279 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:53:32.120648 (+ 369us) client_negotiation.cc:769] Sending connection context
0620 02:53:32.121012 (+ 364us) client_negotiation.cc:240] Negotiation successful
0620 02:53:32.121510 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":298,"threads_started":1}
I20260620 02:53:32.124401 11199 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36143
I20260620 02:53:32.124768 11199 heartbeater.cc:464] Registering TS with master...
I20260620 02:53:32.125597 11199 heartbeater.cc:511] Master 127.0.97.62:36143 requested a full tablet report, sending...
I20260620 02:53:32.127425 10742 ts_manager.cc:194] Registered new tserver with Master: 9879444722a2436a879c65466130b7ec (127.0.97.3:46505)
I20260620 02:53:32.128257 10742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50013
I20260620 02:53:32.132936 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:53:32.151074 11201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:32.137908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49350 (local address 127.0.97.62:36143)
0620 02:53:32.138094 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:32.138102 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:32.138671 (+ 569us) server_negotiation.cc:407] Connection header received
0620 02:53:32.138772 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:32.138776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:32.138840 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:32.138927 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:32.140282 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.141012 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.142557 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.142963 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.143085 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:32.145525 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:32.145531 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:32.145534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:53:32.145567 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:32.148380 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:32.149135 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:32.149139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:32.149142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:32.149261 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:32.149729 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:32.149733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:32.149735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:32.150056 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:53:32.150250 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:32.150641 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 02:53:32.150829 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 02:53:32.155772 10742 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49350:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:53:32.162778 10742 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:53:32.189181 11212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:32.176936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37285 (local address 127.0.0.1:57032)
0620 02:53:32.177735 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:32.177777 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:32.177955 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:32.180295 (+ 2340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:32.180302 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:32.180367 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:53:32.181068 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:32.181092 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.184001 (+ 2909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.184007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:32.186160 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:32.186175 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.188071 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.188077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:32.188284 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:32.188335 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 02:53:32.188546 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 02:53:32.188739 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":58,"spinlock_wait_cycles":5120,"thread_start_us":376,"threads_started":1}
I20260620 02:53:32.189836 11217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:32.178157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55520 (local address 127.0.97.1:34697)
0620 02:53:32.179103 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:32.179112 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:32.179143 (+ 31us) server_negotiation.cc:407] Connection header received
0620 02:53:32.179266 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:32.179277 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:32.179609 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:32.179844 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:53:32.181402 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.182928 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.186737 (+ 3809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.188171 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.188298 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:32.188530 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:32.189303 (+ 773us) server_negotiation.cc:299] Negotiation successful
0620 02:53:32.189499 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":768,"thread_start_us":616,"threads_started":1}
I20260620 02:53:32.189989 11213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:32.177112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57032 (local address 127.0.97.2:37285)
0620 02:53:32.179487 (+ 2375us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:32.179501 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:32.179542 (+ 41us) server_negotiation.cc:407] Connection header received
0620 02:53:32.179658 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:32.179670 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:32.179980 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:32.181841 (+ 1861us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:53:32.182133 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.183771 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.186393 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.187875 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.188014 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:32.188217 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:32.189414 (+ 1197us) server_negotiation.cc:299] Negotiation successful
0620 02:53:32.189609 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1943,"thread_start_us":1836,"threads_started":1}
I20260620 02:53:32.190639 11215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:32.177594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34697 (local address 127.0.0.1:55520)
0620 02:53:32.178346 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:32.178396 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:32.178633 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:32.180195 (+ 1562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:32.180207 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:32.180248 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:53:32.181095 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:32.181113 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.183107 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.183112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:32.186558 (+ 3446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:32.186573 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.188612 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.188617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:32.188885 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:32.188917 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 02:53:32.189029 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 02:53:32.189171 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":152,"spinlock_wait_cycles":1920,"thread_start_us":3707,"threads_started":1}
I20260620 02:53:32.190681 11214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:32.177636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46505 (local address 127.0.0.1:59094)
0620 02:53:32.181548 (+ 3912us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:32.181567 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:32.181662 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:32.182938 (+ 1276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:32.182942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:32.182971 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:53:32.183463 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:32.183475 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.185504 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.185510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:32.187426 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:32.187437 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.189885 (+ 2448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.189891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:32.190076 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:32.190113 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 02:53:32.190205 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 02:53:32.190338 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3789,"thread_start_us":655,"threads_started":1}
I20260620 02:53:32.191404 11216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:32.178137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59094 (local address 127.0.97.3:46505)
0620 02:53:32.182313 (+ 4176us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:32.182321 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:32.182353 (+ 32us) server_negotiation.cc:407] Connection header received
0620 02:53:32.182488 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:32.182497 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:32.182741 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:32.182925 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:53:32.183697 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.185294 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.187578 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.189437 (+ 1859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.189564 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:32.189761 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:32.190927 (+ 1166us) server_negotiation.cc:299] Negotiation successful
0620 02:53:32.191099 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4003,"thread_start_us":3852,"threads_started":1}
I20260620 02:53:32.194118 11000 tablet_service.cc:1467] Processing CreateTablet for tablet 33ee388d017e44d69845abf9db37ac99 (DEFAULT_TABLE table=test-table [id=e3e50ebb51e548a79c361bf982d1656e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:53:32.194502 10866 tablet_service.cc:1467] Processing CreateTablet for tablet 33ee388d017e44d69845abf9db37ac99 (DEFAULT_TABLE table=test-table [id=e3e50ebb51e548a79c361bf982d1656e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:53:32.194537 11134 tablet_service.cc:1467] Processing CreateTablet for tablet 33ee388d017e44d69845abf9db37ac99 (DEFAULT_TABLE table=test-table [id=e3e50ebb51e548a79c361bf982d1656e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:53:32.195480 11000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33ee388d017e44d69845abf9db37ac99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:53:32.195858 10866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33ee388d017e44d69845abf9db37ac99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:53:32.195861 11134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33ee388d017e44d69845abf9db37ac99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:53:32.207707 11218 tablet_bootstrap.cc:492] T 33ee388d017e44d69845abf9db37ac99 P 2e6b52e9ceed4b6db4cccd54f99342af: Bootstrap starting.
I20260620 02:53:32.210486 11219 tablet_bootstrap.cc:492] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e: Bootstrap starting.
I20260620 02:53:32.212165 11220 tablet_bootstrap.cc:492] T 33ee388d017e44d69845abf9db37ac99 P 9879444722a2436a879c65466130b7ec: Bootstrap starting.
I20260620 02:53:32.212718 11218 tablet_bootstrap.cc:654] T 33ee388d017e44d69845abf9db37ac99 P 2e6b52e9ceed4b6db4cccd54f99342af: Neither blocks nor log segments found. Creating new log.
I20260620 02:53:32.213961 11219 tablet_bootstrap.cc:654] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e: Neither blocks nor log segments found. Creating new log.
I20260620 02:53:32.214138 11218 log.cc:824] T 33ee388d017e44d69845abf9db37ac99 P 2e6b52e9ceed4b6db4cccd54f99342af: Log is configured to *not* fsync() on all Append() calls
I20260620 02:53:32.215198 11220 tablet_bootstrap.cc:654] T 33ee388d017e44d69845abf9db37ac99 P 9879444722a2436a879c65466130b7ec: Neither blocks nor log segments found. Creating new log.
I20260620 02:53:32.215293 11219 log.cc:824] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e: Log is configured to *not* fsync() on all Append() calls
I20260620 02:53:32.216377 11220 log.cc:824] T 33ee388d017e44d69845abf9db37ac99 P 9879444722a2436a879c65466130b7ec: Log is configured to *not* fsync() on all Append() calls
I20260620 02:53:32.217317 11218 tablet_bootstrap.cc:492] T 33ee388d017e44d69845abf9db37ac99 P 2e6b52e9ceed4b6db4cccd54f99342af: No bootstrap required, opened a new log
I20260620 02:53:32.217605 11218 ts_tablet_manager.cc:1397] T 33ee388d017e44d69845abf9db37ac99 P 2e6b52e9ceed4b6db4cccd54f99342af: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260620 02:53:32.218232 11219 tablet_bootstrap.cc:492] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e: No bootstrap required, opened a new log
I20260620 02:53:32.218515 11219 ts_tablet_manager.cc:1397] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260620 02:53:32.219094 11220 tablet_bootstrap.cc:492] T 33ee388d017e44d69845abf9db37ac99 P 9879444722a2436a879c65466130b7ec: No bootstrap required, opened a new log
I20260620 02:53:32.219333 11220 ts_tablet_manager.cc:1397] T 33ee388d017e44d69845abf9db37ac99 P 9879444722a2436a879c65466130b7ec: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 02:53:32.227015 11220 raft_consensus.cc:348] T 33ee388d017e44d69845abf9db37ac99 P 9879444722a2436a879c65466130b7ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6b52e9ceed4b6db4cccd54f99342af" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37285 } } peers { permanent_uuid: "c229720ea53a414dab151c7651284d9e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34697 } } peers { permanent_uuid: "9879444722a2436a879c65466130b7ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46505 } }
I20260620 02:53:32.227408 11220 raft_consensus.cc:374] T 33ee388d017e44d69845abf9db37ac99 P 9879444722a2436a879c65466130b7ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:53:32.227501 11220 raft_consensus.cc:729] T 33ee388d017e44d69845abf9db37ac99 P 9879444722a2436a879c65466130b7ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9879444722a2436a879c65466130b7ec, State: Initialized, Role: FOLLOWER
I20260620 02:53:32.228149 11220 consensus_queue.cc:260] T 33ee388d017e44d69845abf9db37ac99 P 9879444722a2436a879c65466130b7ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6b52e9ceed4b6db4cccd54f99342af" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37285 } } peers { permanent_uuid: "c229720ea53a414dab151c7651284d9e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34697 } } peers { permanent_uuid: "9879444722a2436a879c65466130b7ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46505 } }
I20260620 02:53:32.228526 11218 raft_consensus.cc:348] T 33ee388d017e44d69845abf9db37ac99 P 2e6b52e9ceed4b6db4cccd54f99342af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6b52e9ceed4b6db4cccd54f99342af" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37285 } } peers { permanent_uuid: "c229720ea53a414dab151c7651284d9e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34697 } } peers { permanent_uuid: "9879444722a2436a879c65466130b7ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46505 } }
I20260620 02:53:32.229017 11218 raft_consensus.cc:374] T 33ee388d017e44d69845abf9db37ac99 P 2e6b52e9ceed4b6db4cccd54f99342af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:53:32.229122 11218 raft_consensus.cc:729] T 33ee388d017e44d69845abf9db37ac99 P 2e6b52e9ceed4b6db4cccd54f99342af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e6b52e9ceed4b6db4cccd54f99342af, State: Initialized, Role: FOLLOWER
I20260620 02:53:32.229023 11219 raft_consensus.cc:348] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6b52e9ceed4b6db4cccd54f99342af" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37285 } } peers { permanent_uuid: "c229720ea53a414dab151c7651284d9e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34697 } } peers { permanent_uuid: "9879444722a2436a879c65466130b7ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46505 } }
I20260620 02:53:32.229365 11199 heartbeater.cc:503] Master 127.0.97.62:36143 was elected leader, sending a full tablet report...
I20260620 02:53:32.229415 11219 raft_consensus.cc:374] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:53:32.229516 11219 raft_consensus.cc:729] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c229720ea53a414dab151c7651284d9e, State: Initialized, Role: FOLLOWER
I20260620 02:53:32.229820 11218 consensus_queue.cc:260] T 33ee388d017e44d69845abf9db37ac99 P 2e6b52e9ceed4b6db4cccd54f99342af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6b52e9ceed4b6db4cccd54f99342af" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37285 } } peers { permanent_uuid: "c229720ea53a414dab151c7651284d9e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34697 } } peers { permanent_uuid: "9879444722a2436a879c65466130b7ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46505 } }
I20260620 02:53:32.230052 11219 consensus_queue.cc:260] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6b52e9ceed4b6db4cccd54f99342af" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37285 } } peers { permanent_uuid: "c229720ea53a414dab151c7651284d9e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34697 } } peers { permanent_uuid: "9879444722a2436a879c65466130b7ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46505 } }
I20260620 02:53:32.230996 11065 heartbeater.cc:503] Master 127.0.97.62:36143 was elected leader, sending a full tablet report...
I20260620 02:53:32.231853 11220 ts_tablet_manager.cc:1428] T 33ee388d017e44d69845abf9db37ac99 P 9879444722a2436a879c65466130b7ec: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260620 02:53:32.231921 11218 ts_tablet_manager.cc:1428] T 33ee388d017e44d69845abf9db37ac99 P 2e6b52e9ceed4b6db4cccd54f99342af: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260620 02:53:32.234333 11219 ts_tablet_manager.cc:1428] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e: Time spent starting tablet: real 0.016s user 0.013s sys 0.001s
W20260620 02:53:32.258178 10932 tablet.cc:2369] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:53:32.284657 11226 raft_consensus.cc:482] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:53:32.284957 11226 raft_consensus.cc:504] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6b52e9ceed4b6db4cccd54f99342af" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37285 } } peers { permanent_uuid: "c229720ea53a414dab151c7651284d9e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34697 } } peers { permanent_uuid: "9879444722a2436a879c65466130b7ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46505 } }
I20260620 02:53:32.286765 11226 leader_election.cc:288] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e6b52e9ceed4b6db4cccd54f99342af (127.0.97.2:37285), 9879444722a2436a879c65466130b7ec (127.0.97.3:46505)
I20260620 02:53:32.293398 11213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:32.287184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33005 (local address 127.0.97.2:37285)
0620 02:53:32.287499 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:32.287510 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:32.288019 (+ 509us) server_negotiation.cc:407] Connection header received
0620 02:53:32.288245 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:32.288249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:32.288308 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:32.288403 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:53:32.288919 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.289622 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.291581 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.292404 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.292509 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:32.292589 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:32.292993 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 02:53:32.293131 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260620 02:53:32.293407 11216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:32.287302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44747 (local address 127.0.97.3:46505)
0620 02:53:32.287708 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:32.287729 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:32.288334 (+ 605us) server_negotiation.cc:407] Connection header received
0620 02:53:32.288532 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:32.288536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:32.288597 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:32.288691 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:53:32.289234 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.289911 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.291736 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.292566 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.292695 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:32.292797 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:32.293056 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 02:53:32.293181 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234}
I20260620 02:53:32.293447 11227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:32.287022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37285 (local address 127.0.97.1:33005)
0620 02:53:32.287891 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:32.287906 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:32.288091 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:32.288417 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:32.288454 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:32.288479 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:53:32.288780 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:32.288791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.289745 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.289753 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:32.291376 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:32.291389 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.292624 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.292633 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:32.292814 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:32.292853 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 02:53:32.292940 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 02:53:32.293076 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"thread_start_us":532,"threads_started":1}
I20260620 02:53:32.293540 11228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:32.287022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46505 (local address 127.0.97.1:44747)
0620 02:53:32.288229 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0620 02:53:32.288247 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:53:32.288393 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:53:32.288706 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:53:32.288710 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:53:32.288739 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:53:32.289096 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:32.289103 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.290042 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.290050 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:32.291616 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:53:32.291630 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.292760 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.292765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:53:32.292880 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:32.292902 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 02:53:32.293013 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 02:53:32.293143 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1107,"mutex_wait_us":63,"thread_start_us":809,"threads_started":1}
I20260620 02:53:32.294515 11020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ee388d017e44d69845abf9db37ac99" candidate_uuid: "c229720ea53a414dab151c7651284d9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e6b52e9ceed4b6db4cccd54f99342af" is_pre_election: true
I20260620 02:53:32.294651 11154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ee388d017e44d69845abf9db37ac99" candidate_uuid: "c229720ea53a414dab151c7651284d9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9879444722a2436a879c65466130b7ec" is_pre_election: true
I20260620 02:53:32.295045 11020 raft_consensus.cc:2454] T 33ee388d017e44d69845abf9db37ac99 P 2e6b52e9ceed4b6db4cccd54f99342af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c229720ea53a414dab151c7651284d9e in term 0.
I20260620 02:53:32.295194 11154 raft_consensus.cc:2454] T 33ee388d017e44d69845abf9db37ac99 P 9879444722a2436a879c65466130b7ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c229720ea53a414dab151c7651284d9e in term 0.
I20260620 02:53:32.295755 10821 leader_election.cc:302] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e6b52e9ceed4b6db4cccd54f99342af, c229720ea53a414dab151c7651284d9e; no voters:
I20260620 02:53:32.296120 11226 raft_consensus.cc:2781] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:53:32.296271 11226 raft_consensus.cc:482] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:53:32.296401 11226 raft_consensus.cc:3037] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:53:32.298638 11226 raft_consensus.cc:504] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6b52e9ceed4b6db4cccd54f99342af" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37285 } } peers { permanent_uuid: "c229720ea53a414dab151c7651284d9e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34697 } } peers { permanent_uuid: "9879444722a2436a879c65466130b7ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46505 } }
W20260620 02:53:32.299110 11066 tablet.cc:2369] T 33ee388d017e44d69845abf9db37ac99 P 2e6b52e9ceed4b6db4cccd54f99342af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:53:32.299263 11226 leader_election.cc:288] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e [CANDIDATE]: Term 1 election: Requested vote from peers 2e6b52e9ceed4b6db4cccd54f99342af (127.0.97.2:37285), 9879444722a2436a879c65466130b7ec (127.0.97.3:46505)
I20260620 02:53:32.299777 11020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ee388d017e44d69845abf9db37ac99" candidate_uuid: "c229720ea53a414dab151c7651284d9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e6b52e9ceed4b6db4cccd54f99342af"
I20260620 02:53:32.299942 11154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ee388d017e44d69845abf9db37ac99" candidate_uuid: "c229720ea53a414dab151c7651284d9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9879444722a2436a879c65466130b7ec"
I20260620 02:53:32.300000 11020 raft_consensus.cc:3037] T 33ee388d017e44d69845abf9db37ac99 P 2e6b52e9ceed4b6db4cccd54f99342af [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:53:32.300153 11154 raft_consensus.cc:3037] T 33ee388d017e44d69845abf9db37ac99 P 9879444722a2436a879c65466130b7ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:53:32.303340 11154 raft_consensus.cc:2454] T 33ee388d017e44d69845abf9db37ac99 P 9879444722a2436a879c65466130b7ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c229720ea53a414dab151c7651284d9e in term 1.
I20260620 02:53:32.303340 11020 raft_consensus.cc:2454] T 33ee388d017e44d69845abf9db37ac99 P 2e6b52e9ceed4b6db4cccd54f99342af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c229720ea53a414dab151c7651284d9e in term 1.
I20260620 02:53:32.303835 10821 leader_election.cc:302] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2e6b52e9ceed4b6db4cccd54f99342af, 9879444722a2436a879c65466130b7ec, c229720ea53a414dab151c7651284d9e; no voters:
I20260620 02:53:32.304193 11226 raft_consensus.cc:2781] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:53:32.304668 11226 raft_consensus.cc:686] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e [term 1 LEADER]: Becoming Leader. State: Replica: c229720ea53a414dab151c7651284d9e, State: Running, Role: LEADER
I20260620 02:53:32.305220 11226 consensus_queue.cc:237] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6b52e9ceed4b6db4cccd54f99342af" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37285 } } peers { permanent_uuid: "c229720ea53a414dab151c7651284d9e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34697 } } peers { permanent_uuid: "9879444722a2436a879c65466130b7ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46505 } }
I20260620 02:53:32.309707 10741 catalog_manager.cc:5495] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e reported cstate change: term changed from 0 to 1, leader changed from <none> to c229720ea53a414dab151c7651284d9e (127.0.97.1). New cstate: current_term: 1 leader_uuid: "c229720ea53a414dab151c7651284d9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6b52e9ceed4b6db4cccd54f99342af" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c229720ea53a414dab151c7651284d9e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9879444722a2436a879c65466130b7ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46505 } health_report { overall_health: UNKNOWN } } }
W20260620 02:53:32.348989 11200 tablet.cc:2369] T 33ee388d017e44d69845abf9db37ac99 P 9879444722a2436a879c65466130b7ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:53:32.349236 11217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:32.343693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55530 (local address 127.0.97.1:34697)
0620 02:53:32.343875 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:32.343880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:32.343896 (+ 16us) server_negotiation.cc:407] Connection header received
0620 02:53:32.343977 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:32.343980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:32.344064 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:32.344231 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:53:32.344647 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.345384 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.347000 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.347470 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.347582 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:32.348072 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:53:32.348611 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:53:32.348818 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:32.348871 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 02:53:32.348964 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 02:53:32.357825 11020 raft_consensus.cc:1261] T 33ee388d017e44d69845abf9db37ac99 P 2e6b52e9ceed4b6db4cccd54f99342af [term 1 FOLLOWER]: Refusing update from remote peer c229720ea53a414dab151c7651284d9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:53:32.357825 11154 raft_consensus.cc:1261] T 33ee388d017e44d69845abf9db37ac99 P 9879444722a2436a879c65466130b7ec [term 1 FOLLOWER]: Refusing update from remote peer c229720ea53a414dab151c7651284d9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:53:32.358917 11229 consensus_queue.cc:1035] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6b52e9ceed4b6db4cccd54f99342af" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:53:32.359256 11226 consensus_queue.cc:1035] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9879444722a2436a879c65466130b7ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:53:32.371265 11232 mvcc.cc:205] Tried to move back new op lower bound from 7298760754606759936 to 7298760754405998592. Current Snapshot: MvccSnapshot[applied={T|T < 7298760754606759936}]
I20260620 02:53:32.387042 10740 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:53:32.387349 10740 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:53:32.391218 10740 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5818e441e02d48fb9758513ef52a409c: Sending DeleteTablet for 3 replicas of tablet 33ee388d017e44d69845abf9db37ac99
I20260620 02:53:32.392397 11000 tablet_service.cc:1514] Processing DeleteTablet for tablet 33ee388d017e44d69845abf9db37ac99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:53:32 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57032
I20260620 02:53:32.393158 10866 tablet_service.cc:1514] Processing DeleteTablet for tablet 33ee388d017e44d69845abf9db37ac99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:53:32 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55520
I20260620 02:53:32.393314 11241 tablet_replica.cc:330] T 33ee388d017e44d69845abf9db37ac99 P 2e6b52e9ceed4b6db4cccd54f99342af: stopping tablet replica
I20260620 02:53:32.393930 11241 raft_consensus.cc:2229] T 33ee388d017e44d69845abf9db37ac99 P 2e6b52e9ceed4b6db4cccd54f99342af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:53:32.394538 11241 raft_consensus.cc:2258] T 33ee388d017e44d69845abf9db37ac99 P 2e6b52e9ceed4b6db4cccd54f99342af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:53:32.394641 11134 tablet_service.cc:1514] Processing DeleteTablet for tablet 33ee388d017e44d69845abf9db37ac99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:53:32 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59094
I20260620 02:53:32.395542 11243 tablet_replica.cc:330] T 33ee388d017e44d69845abf9db37ac99 P 9879444722a2436a879c65466130b7ec: stopping tablet replica
I20260620 02:53:32.396018 11243 raft_consensus.cc:2229] T 33ee388d017e44d69845abf9db37ac99 P 9879444722a2436a879c65466130b7ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:53:32.396631 11243 raft_consensus.cc:2258] T 33ee388d017e44d69845abf9db37ac99 P 9879444722a2436a879c65466130b7ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:53:32.396786 11241 ts_tablet_manager.cc:1905] T 33ee388d017e44d69845abf9db37ac99 P 2e6b52e9ceed4b6db4cccd54f99342af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:53:32.397274 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 10803
I20260620 02:53:32.397361 11242 tablet_replica.cc:330] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e: stopping tablet replica
I20260620 02:53:32.397727 11242 raft_consensus.cc:2229] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:53:32.398408 11242 raft_consensus.cc:2258] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:53:32.400465 11242 ts_tablet_manager.cc:1905] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:53:32.401360 11243 ts_tablet_manager.cc:1905] T 33ee388d017e44d69845abf9db37ac99 P 9879444722a2436a879c65466130b7ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:53:32.404474 11241 ts_tablet_manager.cc:1918] T 33ee388d017e44d69845abf9db37ac99 P 2e6b52e9ceed4b6db4cccd54f99342af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:53:32.404660 11241 log.cc:1196] T 33ee388d017e44d69845abf9db37ac99 P 2e6b52e9ceed4b6db4cccd54f99342af: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-1/wal/wals/33ee388d017e44d69845abf9db37ac99
I20260620 02:53:32.405169 11241 ts_tablet_manager.cc:1939] T 33ee388d017e44d69845abf9db37ac99 P 2e6b52e9ceed4b6db4cccd54f99342af: Deleting consensus metadata
I20260620 02:53:32.407013 10728 catalog_manager.cc:4841] TS 2e6b52e9ceed4b6db4cccd54f99342af (127.0.97.2:37285): tablet 33ee388d017e44d69845abf9db37ac99 (table test-table [id=e3e50ebb51e548a79c361bf982d1656e]) successfully deleted
I20260620 02:53:32.407557 11242 ts_tablet_manager.cc:1918] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:53:32.407732 11242 log.cc:1196] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-0/wal/wals/33ee388d017e44d69845abf9db37ac99
I20260620 02:53:32.408298 11242 ts_tablet_manager.cc:1939] T 33ee388d017e44d69845abf9db37ac99 P c229720ea53a414dab151c7651284d9e: Deleting consensus metadata
I20260620 02:53:32.409874 10727 catalog_manager.cc:4841] TS c229720ea53a414dab151c7651284d9e (127.0.97.1:34697): tablet 33ee388d017e44d69845abf9db37ac99 (table test-table [id=e3e50ebb51e548a79c361bf982d1656e]) successfully deleted
I20260620 02:53:32.412863 11243 ts_tablet_manager.cc:1918] T 33ee388d017e44d69845abf9db37ac99 P 9879444722a2436a879c65466130b7ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:53:32.413035 11243 log.cc:1196] T 33ee388d017e44d69845abf9db37ac99 P 9879444722a2436a879c65466130b7ec: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781923914418765-388-0/minicluster-data/ts-2/wal/wals/33ee388d017e44d69845abf9db37ac99
I20260620 02:53:32.413573 11243 ts_tablet_manager.cc:1939] T 33ee388d017e44d69845abf9db37ac99 P 9879444722a2436a879c65466130b7ec: Deleting consensus metadata
I20260620 02:53:32.415216 10726 catalog_manager.cc:4841] TS 9879444722a2436a879c65466130b7ec (127.0.97.3:46505): tablet 33ee388d017e44d69845abf9db37ac99 (table test-table [id=e3e50ebb51e548a79c361bf982d1656e]) successfully deleted
Jun 20 02:53:32 dist-test-slave-8p8c krb5kdc[10697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:53:32.427181 11217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:32.398356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55534 (local address 127.0.97.1:34697)
0620 02:53:32.398577 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:32.398583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:32.407890 (+ 9307us) server_negotiation.cc:407] Connection header received
0620 02:53:32.408111 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:32.408117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:32.408217 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:32.408339 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:32.410121 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.411384 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.413246 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.413974 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.414119 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:32.419035 (+ 4916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:32.419047 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:32.419060 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:53:32.419101 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:32.423217 (+ 4116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:32.423920 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:32.423931 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:32.423936 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:32.424161 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:32.424604 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:32.424610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:32.424613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:32.425048 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:53:32.425391 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:32.426419 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0620 02:53:32.426729 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 02:53:32.427989 10926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:53:32.528961 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 10803
I20260620 02:53:32.543771 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 10937
Jun 20 02:53:32 dist-test-slave-8p8c krb5kdc[10697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:53:32.559398 11213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:32.544597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57042 (local address 127.0.97.2:37285)
0620 02:53:32.544821 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:32.544826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:32.544845 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:53:32.545010 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:32.545015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:32.545085 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:32.545192 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:32.546428 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.547213 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.548442 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.548854 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.548995 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:32.553461 (+ 4466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:32.553472 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:32.553486 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:53:32.553519 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:32.556530 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:32.557286 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:32.557293 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:32.557295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:32.557417 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:32.557799 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:32.557808 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:32.557810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:32.558186 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:53:32.558501 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:32.558810 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 02:53:32.559026 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 02:53:32.560119 11060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:53:32.649605 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 10937
I20260620 02:53:32.666332 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 11071
Jun 20 02:53:32 dist-test-slave-8p8c krb5kdc[10697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:53:32.681878 11216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:53:32.667205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59100 (local address 127.0.97.3:46505)
0620 02:53:32.667440 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 02:53:32.667448 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:53:32.667473 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:53:32.667628 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:53:32.667637 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:53:32.667734 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:53:32.667855 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:53:32.669156 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.669902 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.671270 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:53:32.671808 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:53:32.671926 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:53:32.676234 (+ 4308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:53:32.676250 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:53:32.676266 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:53:32.676300 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:53:32.679256 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:32.679903 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:32.679912 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:32.679914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:32.680076 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:53:32.680546 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:53:32.680550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:53:32.680552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:53:32.680908 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:53:32.681130 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 02:53:32.681472 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 02:53:32.681639 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 02:53:32.682595 11194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:53:32.772825 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 11071
I20260620 02:53:32.789769 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 10710
I20260620 02:53:32.791064 10772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:53:32.900696 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 10710
2026-06-20T02:53:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4324 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:53:33 dist-test-slave-8p8c krb5kdc[11251](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:53:33 dist-test-slave-8p8c krb5kdc[11251](info): set up 2 sockets
Jun 20 02:53:33 dist-test-slave-8p8c krb5kdc[11251](info): commencing operation
krb5kdc: starting...
W20260620 02:53:34.011488 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:53:34 dist-test-slave-8p8c krb5kdc[11251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:53:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:53:34Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260620 02:53:34.342254 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskgxGzum/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/postgres -l logfile start
2026-06-20 02:53:37.206 UTC [11281] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 02:53:37.206 UTC [11281] LOG: listening on IPv4 address "127.0.97.20", port 34277
2026-06-20 02:53:37.210 UTC [11281] LOG: listening on Unix socket "/tmp/.s.PGSQL.34277"
2026-06-20 02:53:37.233 UTC [11284] LOG: database system was shut down at 2026-06-20 02:53:35 UTC
2026-06-20 02:53:37.237 UTC [11281] LOG: database system is ready to accept connections
I20260620 02:53:38.223769 388 mini_postgres.cc:96] Postgres bound to 34277
2026-06-20 02:53:38.242 UTC [11292] FATAL: database "slave" does not exist
127.0.97.20:34277 - accepting connections
I20260620 02:53:38.243346 388 mini_ranger.cc:161] Starting Ranger...
I20260620 02:53:38.271032 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 02:53:38.877952 388 mini_ranger.cc:87] Created ranger Postgres database
I20260620 02:53:38.878098 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ranger-admin
2026-06-20 02:53:39,186 [I] DB FLAVOR :POSTGRES
2026-06-20 02:53:39,187 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 02:53:39,187 [I] Checking connection..
2026-06-20 02:53:39,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 02:53:39,439 [I] Checking connection passed.
2026-06-20 02:53:39,440 [I] --------- Verifying version history table ---------
2026-06-20 02:53:39,440 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 02:53:39,703 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 02:53:39,703 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 02:53:39,949 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 02:53:39,950 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 02:53:39,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34277/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 02:53:40.178 UTC [11356] WARNING: there is no transaction in progress
2026-06-20 02:53:40,185 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 02:53:40,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 02:53:40,452 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 02:53:40,452 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 02:53:40,452 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 02:53:40,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 02:53:40,966 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8p8c.c.gcp-upstream.internal','N') ;"
2026-06-20 02:53:41,202 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 02:53:41,459 [I] Table x_portal_user does not exist in database ranger
2026-06-20 02:53:41,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 02:53:41,710 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 02:53:41,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 02:53:41,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 02:53:42,237 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 02:53:42,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34277/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 02:53:43.271 UTC [11464] WARNING: there is no transaction in progress
2026-06-20 02:53:43.288 UTC [11464] WARNING: there is no transaction in progress
2026-06-20 02:53:43.305 UTC [11464] WARNING: there is no transaction in progress
2026-06-20 02:53:43.321 UTC [11464] WARNING: there is no transaction in progress
2026-06-20 02:53:43.337 UTC [11464] WARNING: there is no transaction in progress
2026-06-20 02:53:43.359 UTC [11464] WARNING: there is no transaction in progress
2026-06-20 02:53:43.425 UTC [11464] WARNING: there is no transaction in progress
2026-06-20 02:53:43.432 UTC [11464] WARNING: there is no transaction in progress
2026-06-20 02:53:43.447 UTC [11464] WARNING: there is no transaction in progress
2026-06-20 02:53:43.458 UTC [11464] WARNING: there is no transaction in progress
2026-06-20 02:53:43.469 UTC [11464] WARNING: there is no transaction in progress
2026-06-20 02:53:43.484 UTC [11464] WARNING: there is no transaction in progress
2026-06-20 02:53:43.495 UTC [11464] WARNING: there is no transaction in progress
2026-06-20 02:53:43.503 UTC [11464] WARNING: there is no transaction in progress
2026-06-20 02:53:43.512 UTC [11464] WARNING: there is no transaction in progress
2026-06-20 02:53:43.522 UTC [11464] WARNING: there is no transaction in progress
2026-06-20 02:53:44,321 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 02:53:44,322 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 02:53:44,552 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 02:53:44,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 02:53:44,806 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 02:53:44,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 02:53:45,070 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 02:53:45,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 02:53:45,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 02:53:45,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8p8c.c.gcp-upstream.internal';"
2026-06-20 02:53:45,797 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 02:53:45,797 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 02:53:46,050 [I] DB_PATCHES have already been applied
I20260620 02:53:46.056799 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskgxGzum/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 02:53:46.056890 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260620 02:53:46.071610 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:35677
Jun 20, 2026 2:53:46 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:53:46 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:53:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 2:53:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 2:53:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 2:53:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 2:53:46 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 02:53:47.229389 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.986s user 0.009s sys 0.031s
Jun 20, 2026 2:54:06 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 2:54:06 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 2:54:06 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 2:54:07 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 2:54:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 2:54:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 2:54:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 2:54:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 2:54:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 2:54:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 2:54:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 2:54:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 2:54:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 2:54:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 2:54:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 2:54:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 2:54:07 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 2:54:07 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 02:54:08.516215 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 02:54:08 dist-test-slave-8p8c krb5kdc[11251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924048, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 02:54:08.684734 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 02:54:08.707080 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 02:54:09.898310 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 02:54:09.898504 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ranger-kms
I20260620 02:54:09.898671 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:34277
2026-06-20 02:54:10,309 [I] DB FLAVOR :POSTGRES
2026-06-20 02:54:10,309 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 02:54:10,309 [I] Checking connection
2026-06-20 02:54:10,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34277/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 02:54:10,571 [I] connection success
2026-06-20 02:54:10,571 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 02:54:10,572 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 02:54:10,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34277/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 02:54:10,801 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 02:54:10,802 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 02:54:10,802 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 02:54:10,802 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34277/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 02:54:11,083 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 02:54:11.578967 388 mini_ranger_kms.cc:310] Created kms service
I20260620 02:54:11.771247 388 mini_ranger_kms.cc:326] Created kudu user
I20260620 02:54:11.815869 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 02:54:11 dist-test-slave-8p8c krb5kdc[11251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924048, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:54:12.266039 388 mini_ranger_kms.cc:384] Added ranger policy
I20260620 02:54:12.266433 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 02:54:12.266512 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260620 02:54:12.287629 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11644
I20260620 02:54:12.287895 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:45953
Jun 20, 2026 2:54:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:54:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:54:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 2:54:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 2:54:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 2:54:13 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 02:54:14 dist-test-slave-8p8c krb5kdc[11251](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781924054, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:54:14.668977 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.984s user 0.114s sys 0.318s
I20260620 02:54:14.669292 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 02:54:14.669517 388 mini_ranger_kms.cc:401] 127.0.97.20:45953/kms/v1/keys
Jun 20 02:54:15 dist-test-slave-8p8c krb5kdc[11251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924054, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 02:54:15 dist-test-slave-8p8c krb5kdc[11251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924054, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 02:54:17 dist-test-slave-8p8c krb5kdc[11251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:54:17.781071 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41507
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41507
--ranger_config_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45953/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:54:18.142161 11695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:54:18.142570 11695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:54:18.142748 11695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:54:18.150216 11695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:54:18.150424 11695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:54:18.150553 11695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:54:18.150664 11695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:54:18.150771 11695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:54:18.163232 11695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:45953/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41507
--ranger_config_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:54:18.164860 11695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:54:18.166545 11695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:54:18.180421 11700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:54:18.180508 11695 server_base.cc:1034] running on GCE node
W20260620 02:54:18.180369 11701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:54:18.180409 11703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:54:18.181840 11695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:54:18.183444 11695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:54:18.184697 11695 hybrid_clock.cc:648] HybridClock initialized: now 1781924058184653 us; error 37 us; skew 500 ppm
I20260620 02:54:18.185081 11695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:54:18 dist-test-slave-8p8c krb5kdc[11251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:54:18.191972 11695 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:54:18.193612 11695 webserver.cc:460] Webserver started at http://127.0.97.62:44407/ using document root <none> and password file <none>
I20260620 02:54:18.194685 11695 fs_manager.cc:359] Metadata directory not provided
I20260620 02:54:18.194787 11695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:54:18.195041 11695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 02:54:18 dist-test-slave-8p8c krb5kdc[11251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:54:18.256654 11695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "4b7211d166be4d64b99566cf68f7e40b"
format_stamp: "Formatted at 2026-06-20 02:54:18 on dist-test-slave-8p8c"
server_key: "7246e3779851503605de7380a6d0bb40"
server_key_iv: "e9ed368bbc9326d29b7b78c9d17c86e9"
server_key_version: "kuduclusterkey@0"
I20260620 02:54:18.257745 11695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "4b7211d166be4d64b99566cf68f7e40b"
format_stamp: "Formatted at 2026-06-20 02:54:18 on dist-test-slave-8p8c"
server_key: "7246e3779851503605de7380a6d0bb40"
server_key_iv: "e9ed368bbc9326d29b7b78c9d17c86e9"
server_key_version: "kuduclusterkey@0"
I20260620 02:54:18.265156 11695 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260620 02:54:18.286450 11710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:54:18.288266 11695 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 02:54:18.288457 11695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "4b7211d166be4d64b99566cf68f7e40b"
format_stamp: "Formatted at 2026-06-20 02:54:18 on dist-test-slave-8p8c"
server_key: "7246e3779851503605de7380a6d0bb40"
server_key_iv: "e9ed368bbc9326d29b7b78c9d17c86e9"
server_key_version: "kuduclusterkey@0"
I20260620 02:54:18.288726 11695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:54:18.306090 11695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:54:18.310679 11695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:54:18.310940 11695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:54:18.332146 11695 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41507
I20260620 02:54:18.332187 11761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41507 every 8 connection(s)
I20260620 02:54:18.333498 11695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:54:18.337327 11762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:54:18.339851 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 11695
I20260620 02:54:18.340059 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:54:18.340885 388 external_mini_cluster.cc:1428] Setting key 586cc95db27b7a1c2ff459aa8cfa916a
I20260620 02:54:18.349720 11762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b7211d166be4d64b99566cf68f7e40b: Bootstrap starting.
I20260620 02:54:18.353123 11762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b7211d166be4d64b99566cf68f7e40b: Neither blocks nor log segments found. Creating new log.
I20260620 02:54:18.354204 11762 log.cc:824] T 00000000000000000000000000000000 P 4b7211d166be4d64b99566cf68f7e40b: Log is configured to *not* fsync() on all Append() calls
Jun 20 02:54:18 dist-test-slave-8p8c krb5kdc[11251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:54:18.357391 11762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b7211d166be4d64b99566cf68f7e40b: No bootstrap required, opened a new log
I20260620 02:54:18.364482 11765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:54:18.343070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37286 (local address 127.0.97.62:41507)
0620 02:54:18.343899 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0620 02:54:18.343911 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:54:18.343950 (+ 39us) server_negotiation.cc:407] Connection header received
0620 02:54:18.345976 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:54:18.345998 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:54:18.346442 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:54:18.346874 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:54:18.348181 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:54:18.349421 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:54:18.350773 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:54:18.351436 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:54:18.351642 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:54:18.355947 (+ 4305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:54:18.355954 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:54:18.355989 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:54:18.356021 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:54:18.360001 (+ 3980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:54:18.360958 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:54:18.360964 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:54:18.360970 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:54:18.361148 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:54:18.361631 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:54:18.361634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:54:18.361637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:54:18.362215 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:54:18.362524 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 02:54:18.363045 (+ 521us) server_negotiation.cc:299] Negotiation successful
0620 02:54:18.363469 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":365,"threads_started":1}
I20260620 02:54:18.366937 11762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b7211d166be4d64b99566cf68f7e40b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7211d166be4d64b99566cf68f7e40b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41507 } }
I20260620 02:54:18.367311 11762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b7211d166be4d64b99566cf68f7e40b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:54:18.367453 11762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b7211d166be4d64b99566cf68f7e40b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7211d166be4d64b99566cf68f7e40b, State: Initialized, Role: FOLLOWER
I20260620 02:54:18.368000 11762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b7211d166be4d64b99566cf68f7e40b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7211d166be4d64b99566cf68f7e40b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41507 } }
I20260620 02:54:18.368238 11762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b7211d166be4d64b99566cf68f7e40b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:54:18.368379 11762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b7211d166be4d64b99566cf68f7e40b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:54:18.368542 11762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b7211d166be4d64b99566cf68f7e40b [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:54:18.370688 11762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b7211d166be4d64b99566cf68f7e40b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7211d166be4d64b99566cf68f7e40b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41507 } }
I20260620 02:54:18.371155 11762 leader_election.cc:302] T 00000000000000000000000000000000 P 4b7211d166be4d64b99566cf68f7e40b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b7211d166be4d64b99566cf68f7e40b; no voters:
I20260620 02:54:18.371783 11762 leader_election.cc:288] T 00000000000000000000000000000000 P 4b7211d166be4d64b99566cf68f7e40b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:54:18.371869 11767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b7211d166be4d64b99566cf68f7e40b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:54:18.373564 11767 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b7211d166be4d64b99566cf68f7e40b [term 1 LEADER]: Becoming Leader. State: Replica: 4b7211d166be4d64b99566cf68f7e40b, State: Running, Role: LEADER
I20260620 02:54:18.374099 11767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b7211d166be4d64b99566cf68f7e40b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7211d166be4d64b99566cf68f7e40b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41507 } }
I20260620 02:54:18.375795 11768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7211d166be4d64b99566cf68f7e40b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b7211d166be4d64b99566cf68f7e40b. Latest consensus state: current_term: 1 leader_uuid: "4b7211d166be4d64b99566cf68f7e40b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7211d166be4d64b99566cf68f7e40b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41507 } } }
I20260620 02:54:18.376199 11768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7211d166be4d64b99566cf68f7e40b [sys.catalog]: This master's current role is: LEADER
I20260620 02:54:18.376266 11762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b7211d166be4d64b99566cf68f7e40b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:54:18.376695 11769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7211d166be4d64b99566cf68f7e40b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b7211d166be4d64b99566cf68f7e40b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7211d166be4d64b99566cf68f7e40b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41507 } } }
I20260620 02:54:18.377025 11769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7211d166be4d64b99566cf68f7e40b [sys.catalog]: This master's current role is: LEADER
I20260620 02:54:18.378968 11771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:54:18.384863 11771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:54:18.386570 11762 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 02:54:18.405375 11771 catalog_manager.cc:1347] Generated new cluster ID: d627abae7d214731a3b6bf7849ceb485
I20260620 02:54:18.405663 11771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:54:18.432844 11771 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:54:18.435159 11771 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:54:18.445456 11771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b7211d166be4d64b99566cf68f7e40b: Generated new TSK 0
I20260620 02:54:18.446693 11771 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:54:19.227730 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45953/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41507
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:54:19 dist-test-slave-8p8c krb5kdc[11251](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781924059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 02:54:19.662669 11807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:54:19.662943 11807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:54:19.663107 11807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:54:19.671307 11807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:54:19.671509 11807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:54:19.671770 11807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:54:19.685233 11807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:45953/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:54:19.687254 11807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:54:19.689013 11807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:54:19.710243 11818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:54:19.714730 11817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:54:19.715847 11807 server_base.cc:1034] running on GCE node
W20260620 02:54:19.716223 11820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:54:19.716979 11807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:54:19.718638 11807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:54:19.720340 11807 hybrid_clock.cc:648] HybridClock initialized: now 1781924059720220 us; error 104 us; skew 500 ppm
I20260620 02:54:19.720717 11807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:54:19 dist-test-slave-8p8c krb5kdc[11251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:54:19.724821 11807 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:54:19.726819 11807 webserver.cc:460] Webserver started at http://127.0.97.1:41513/ using document root <none> and password file <none>
I20260620 02:54:19.727697 11807 fs_manager.cc:359] Metadata directory not provided
I20260620 02:54:19.727835 11807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:54:19.728241 11807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 02:54:19 dist-test-slave-8p8c krb5kdc[11251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:54:19.749450 11807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-0/data/instance:
uuid: "7e870a45402841deb26401373042cf0e"
format_stamp: "Formatted at 2026-06-20 02:54:19 on dist-test-slave-8p8c"
server_key: "9d405b316c9407928448c27f5e9aae54"
server_key_iv: "279a6893f72eb03241aeb974dd5f18a1"
server_key_version: "kuduclusterkey@0"
I20260620 02:54:19.751253 11807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7e870a45402841deb26401373042cf0e"
format_stamp: "Formatted at 2026-06-20 02:54:19 on dist-test-slave-8p8c"
server_key: "9d405b316c9407928448c27f5e9aae54"
server_key_iv: "279a6893f72eb03241aeb974dd5f18a1"
server_key_version: "kuduclusterkey@0"
I20260620 02:54:19.772542 11807 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.008s sys 0.011s
I20260620 02:54:19.797260 11827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:54:19.799371 11807 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 02:54:19.799583 11807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-0/wal
uuid: "7e870a45402841deb26401373042cf0e"
format_stamp: "Formatted at 2026-06-20 02:54:19 on dist-test-slave-8p8c"
server_key: "9d405b316c9407928448c27f5e9aae54"
server_key_iv: "279a6893f72eb03241aeb974dd5f18a1"
server_key_version: "kuduclusterkey@0"
I20260620 02:54:19.799844 11807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:54:19.822651 11807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:54:19.828538 11807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:54:19.828914 11807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:54:19.835340 11807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:54:19.838117 11807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:54:19.838276 11807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:54:19.838492 11807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:54:19.838639 11807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:54:19.931144 11807 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40957
I20260620 02:54:19.932675 11939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40957 every 8 connection(s)
I20260620 02:54:19.933087 11807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:54:19.939007 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 11807
I20260620 02:54:19.939221 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:54:19.940049 388 external_mini_cluster.cc:1428] Setting key b76a711b46be2db8ae62e85574b0847e
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Jun 20 02:54:19 dist-test-slave-8p8c krb5kdc[11251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:54:19.998260 11942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:54:19.951600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39823 (local address 127.0.97.62:41507)
0620 02:54:19.971020 (+ 19420us) server_negotiation.cc:206] Beginning negotiation
0620 02:54:19.971030 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:54:19.971072 (+ 42us) server_negotiation.cc:407] Connection header received
0620 02:54:19.971153 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:54:19.971161 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:54:19.971280 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:54:19.971450 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:54:19.973907 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:54:19.975152 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:54:19.977776 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:54:19.978484 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:54:19.978628 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:54:19.983508 (+ 4880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:54:19.983515 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:54:19.983519 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:54:19.983561 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:54:19.991778 (+ 8217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:54:19.993703 (+ 1925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:54:19.993709 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:54:19.993712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:54:19.993880 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:54:19.994480 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:54:19.994485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:54:19.994488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:54:19.994854 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:54:19.994983 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 02:54:19.997393 (+ 2410us) server_negotiation.cc:299] Negotiation successful
0620 02:54:19.997874 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19260,"thread_start_us":19131,"threads_started":1}
I20260620 02:54:19.999895 11943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:54:19.952470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41507 (local address 127.0.97.1:39823)
0620 02:54:19.963766 (+ 11296us) negotiation.cc:107] Waiting for socket to connect
0620 02:54:19.963853 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0620 02:54:19.968644 (+ 4791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:54:19.971873 (+ 3229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:54:19.971888 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:54:19.972692 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:54:19.973622 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:54:19.973661 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:54:19.975366 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:54:19.975372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:54:19.977549 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:54:19.977586 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:54:19.978749 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:54:19.978759 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:54:19.979101 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:54:19.980011 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:54:19.980045 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:54:19.983172 (+ 3127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:54:19.992078 (+ 8906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:54:19.992090 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:54:19.992111 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:54:19.993508 (+ 1397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:54:19.994029 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:54:19.994034 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:54:19.994038 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:54:19.994297 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:54:19.995068 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:54:19.995079 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:54:19.995562 (+ 483us) client_negotiation.cc:769] Sending connection context
0620 02:54:19.995880 (+ 318us) client_negotiation.cc:240] Negotiation successful
0620 02:54:19.996468 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10772,"thread_start_us":13728,"threads_started":1}
I20260620 02:54:20.003002 11940 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41507
I20260620 02:54:20.003386 11940 heartbeater.cc:464] Registering TS with master...
I20260620 02:54:20.004343 11940 heartbeater.cc:511] Master 127.0.97.62:41507 requested a full tablet report, sending...
I20260620 02:54:20.007169 11727 ts_manager.cc:194] Registered new tserver with Master: 7e870a45402841deb26401373042cf0e (127.0.97.1:40957)
I20260620 02:54:20.009354 11727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39823
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:54:20.022197 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45953/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41507
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:54:20 dist-test-slave-8p8c krb5kdc[11251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 02:54:20 dist-test-slave-8p8c krb5kdc[11251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260620 02:54:20.539819 11946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:54:20.540342 11946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:54:20.540515 11946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:54:20.554138 11946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:54:20.554412 11946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:54:20.554682 11946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:54:20.571272 11946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:45953/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:54:20.573573 11946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:54:20.575560 11946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:54:20.598627 11953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:54:20.600654 11954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:54:20.607569 11956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:54:20.608886 11946 server_base.cc:1034] running on GCE node
I20260620 02:54:20.609783 11946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:54:20.614691 11946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:54:20.616050 11946 hybrid_clock.cc:648] HybridClock initialized: now 1781924060615924 us; error 110 us; skew 500 ppm
I20260620 02:54:20.616493 11946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:54:20 dist-test-slave-8p8c krb5kdc[11251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:54:20.621219 11946 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:54:20.623437 11946 webserver.cc:460] Webserver started at http://127.0.97.2:35319/ using document root <none> and password file <none>
I20260620 02:54:20.624455 11946 fs_manager.cc:359] Metadata directory not provided
I20260620 02:54:20.624603 11946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:54:20.624989 11946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 02:54:20 dist-test-slave-8p8c krb5kdc[11251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:54:20.648185 11946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-1/data/instance:
uuid: "e9d5c598d2df4cb5bffba30363c51ff2"
format_stamp: "Formatted at 2026-06-20 02:54:20 on dist-test-slave-8p8c"
server_key: "ee8cbb6061bd8e4272497f018863b9da"
server_key_iv: "d201b4798821950c2df9176fe011ac4b"
server_key_version: "kuduclusterkey@0"
I20260620 02:54:20.649339 11946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e9d5c598d2df4cb5bffba30363c51ff2"
format_stamp: "Formatted at 2026-06-20 02:54:20 on dist-test-slave-8p8c"
server_key: "ee8cbb6061bd8e4272497f018863b9da"
server_key_iv: "d201b4798821950c2df9176fe011ac4b"
server_key_version: "kuduclusterkey@0"
I20260620 02:54:20.657261 11946 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260620 02:54:20.679128 11970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:54:20.681329 11946 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.005s
I20260620 02:54:20.681537 11946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-1/wal
uuid: "e9d5c598d2df4cb5bffba30363c51ff2"
format_stamp: "Formatted at 2026-06-20 02:54:20 on dist-test-slave-8p8c"
server_key: "ee8cbb6061bd8e4272497f018863b9da"
server_key_iv: "d201b4798821950c2df9176fe011ac4b"
server_key_version: "kuduclusterkey@0"
I20260620 02:54:20.681777 11946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:54:20.697513 11946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:54:20.702145 11946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:54:20.702507 11946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:54:20.703508 11946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:54:20.704921 11946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:54:20.705000 11946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:54:20.705150 11946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:54:20.705220 11946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:54:20.741361 11946 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36689
I20260620 02:54:20.741408 12082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36689 every 8 connection(s)
I20260620 02:54:20.742692 11946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:54:20.752213 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 11946
I20260620 02:54:20.752434 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:54:20.752929 388 external_mini_cluster.cc:1428] Setting key c4a6914a4b97a4685863552ba24993f0
Jun 20 02:54:20 dist-test-slave-8p8c krb5kdc[11251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:54:20.765549 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:54:20.745425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51073 (local address 127.0.97.62:41507)
0620 02:54:20.745896 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0620 02:54:20.745901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:54:20.747004 (+ 1103us) server_negotiation.cc:407] Connection header received
0620 02:54:20.748885 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:54:20.748891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:54:20.748975 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:54:20.749097 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:54:20.750967 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:54:20.751705 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:54:20.753417 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:54:20.753992 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:54:20.754148 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:54:20.757957 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:54:20.757966 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:54:20.757974 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:54:20.758020 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:54:20.761790 (+ 3770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:54:20.762744 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:54:20.762752 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:54:20.762757 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:54:20.762917 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:54:20.763476 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:54:20.763483 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:54:20.763485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:54:20.763888 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:54:20.764019 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 02:54:20.765078 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0620 02:54:20.765292 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":267,"threads_started":1}
I20260620 02:54:20.767119 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:54:20.745901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41507 (local address 127.0.97.2:51073)
0620 02:54:20.746842 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0620 02:54:20.746888 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 02:54:20.748550 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:54:20.749328 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:54:20.749337 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:54:20.750147 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:54:20.750795 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:54:20.750814 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:54:20.751883 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:54:20.751887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:54:20.753271 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:54:20.753291 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:54:20.754229 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:54:20.754239 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:54:20.754535 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:54:20.755322 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:54:20.755347 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:54:20.757717 (+ 2370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:54:20.761952 (+ 4235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:54:20.761960 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:54:20.761975 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:54:20.762564 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:54:20.763058 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:54:20.763062 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:54:20.763064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:54:20.763298 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:54:20.764138 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:54:20.764148 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:54:20.764633 (+ 485us) client_negotiation.cc:769] Sending connection context
0620 02:54:20.764947 (+ 314us) client_negotiation.cc:240] Negotiation successful
0620 02:54:20.765542 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":311,"threads_started":1}
I20260620 02:54:20.768760 12083 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41507
I20260620 02:54:20.769093 12083 heartbeater.cc:464] Registering TS with master...
I20260620 02:54:20.769848 12083 heartbeater.cc:511] Master 127.0.97.62:41507 requested a full tablet report, sending...
I20260620 02:54:20.771479 11727 ts_manager.cc:194] Registered new tserver with Master: e9d5c598d2df4cb5bffba30363c51ff2 (127.0.97.2:36689)
I20260620 02:54:20.772806 11727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51073
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:54:20.810830 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45953/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41507
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:54:21.013837 11940 heartbeater.cc:503] Master 127.0.97.62:41507 was elected leader, sending a full tablet report...
W20260620 02:54:21.161834 12089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:54:21.162186 12089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:54:21.162348 12089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:54:21.171169 12089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:54:21.171335 12089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:54:21.171602 12089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:54:21.182957 12089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:45953/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:54:21.184352 12089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:54:21.185801 12089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:54:21.201164 12095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:54:21.201352 12094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:54:21.201315 12097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:54:21.201849 12089 server_base.cc:1034] running on GCE node
I20260620 02:54:21.202481 12089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:54:21.203420 12089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:54:21.204772 12089 hybrid_clock.cc:648] HybridClock initialized: now 1781924061204715 us; error 33 us; skew 500 ppm
I20260620 02:54:21.205112 12089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:54:21 dist-test-slave-8p8c krb5kdc[11251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924061, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:54:21.208505 12089 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:54:21.210055 12089 webserver.cc:460] Webserver started at http://127.0.97.3:33729/ using document root <none> and password file <none>
I20260620 02:54:21.210814 12089 fs_manager.cc:359] Metadata directory not provided
I20260620 02:54:21.210959 12089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:54:21.211280 12089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 02:54:21 dist-test-slave-8p8c krb5kdc[11251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924061, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:54:21.228910 12089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-2/data/instance:
uuid: "88ecaa37705c4e1a8be53e56c63231c7"
format_stamp: "Formatted at 2026-06-20 02:54:21 on dist-test-slave-8p8c"
server_key: "d5563252765641e12983fbdebc2e9c2b"
server_key_iv: "2d680f17ad3b44febafd627e23fdbfb8"
server_key_version: "kuduclusterkey@0"
I20260620 02:54:21.230187 12089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-2/wal/instance:
uuid: "88ecaa37705c4e1a8be53e56c63231c7"
format_stamp: "Formatted at 2026-06-20 02:54:21 on dist-test-slave-8p8c"
server_key: "d5563252765641e12983fbdebc2e9c2b"
server_key_iv: "2d680f17ad3b44febafd627e23fdbfb8"
server_key_version: "kuduclusterkey@0"
I20260620 02:54:21.236827 12089 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260620 02:54:21.253665 12104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:54:21.255425 12089 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260620 02:54:21.255626 12089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-2/wal
uuid: "88ecaa37705c4e1a8be53e56c63231c7"
format_stamp: "Formatted at 2026-06-20 02:54:21 on dist-test-slave-8p8c"
server_key: "d5563252765641e12983fbdebc2e9c2b"
server_key_iv: "2d680f17ad3b44febafd627e23fdbfb8"
server_key_version: "kuduclusterkey@0"
I20260620 02:54:21.255867 12089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:54:21.271976 12089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:54:21.276396 12089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:54:21.276705 12089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:54:21.277850 12089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:54:21.279251 12089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:54:21.279353 12089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:54:21.279491 12089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:54:21.279980 12089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:54:21.315508 12089 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39709
I20260620 02:54:21.315542 12216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39709 every 8 connection(s)
I20260620 02:54:21.316803 12089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:54:21.324913 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 12089
I20260620 02:54:21.325102 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:54:21.325564 388 external_mini_cluster.cc:1428] Setting key ff7c18785c7c6bcb03a9d1f49604b601
Jun 20 02:54:21 dist-test-slave-8p8c krb5kdc[11251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924061, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:54:21.338629 12219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:54:21.319577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35585 (local address 127.0.97.62:41507)
0620 02:54:21.320035 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0620 02:54:21.320040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:54:21.320771 (+ 731us) server_negotiation.cc:407] Connection header received
0620 02:54:21.322667 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:54:21.322673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:54:21.322780 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:54:21.322895 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:54:21.324296 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.325080 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.327015 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.327456 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.327598 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:54:21.331863 (+ 4265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:54:21.331870 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:54:21.331874 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:54:21.331914 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:54:21.335453 (+ 3539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:54:21.336223 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:54:21.336228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:54:21.336232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:54:21.336406 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:54:21.336848 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:54:21.336854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:54:21.336857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:54:21.337224 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:54:21.337357 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 02:54:21.338019 (+ 662us) server_negotiation.cc:299] Negotiation successful
0620 02:54:21.338228 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":249,"threads_started":1}
I20260620 02:54:21.339530 12220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:54:21.319922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41507 (local address 127.0.97.3:35585)
0620 02:54:21.320607 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0620 02:54:21.320645 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 02:54:21.322304 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:54:21.323150 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:54:21.323159 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:54:21.323561 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:54:21.324123 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:54:21.324140 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.325271 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.325274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:54:21.326872 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:54:21.326889 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.327673 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.327682 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:54:21.327942 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:54:21.328810 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:54:21.328835 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:54:21.331624 (+ 2789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:54:21.335613 (+ 3989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:54:21.335620 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:54:21.335635 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:54:21.336094 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:54:21.336516 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:54:21.336520 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:54:21.336522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:54:21.336738 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:54:21.337343 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:54:21.337349 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:54:21.337670 (+ 321us) client_negotiation.cc:769] Sending connection context
0620 02:54:21.337892 (+ 222us) client_negotiation.cc:240] Negotiation successful
0620 02:54:21.338277 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":279,"threads_started":1}
I20260620 02:54:21.341310 12217 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41507
I20260620 02:54:21.341665 12217 heartbeater.cc:464] Registering TS with master...
I20260620 02:54:21.342530 12217 heartbeater.cc:511] Master 127.0.97.62:41507 requested a full tablet report, sending...
I20260620 02:54:21.344142 11726 ts_manager.cc:194] Registered new tserver with Master: 88ecaa37705c4e1a8be53e56c63231c7 (127.0.97.3:39709)
I20260620 02:54:21.345151 11726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35585
I20260620 02:54:21.354548 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:54:21.374895 12219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:54:21.359663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37288 (local address 127.0.97.62:41507)
0620 02:54:21.359927 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 02:54:21.359936 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:54:21.360317 (+ 381us) server_negotiation.cc:407] Connection header received
0620 02:54:21.360593 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:54:21.360597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:54:21.360674 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:54:21.360788 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:54:21.362261 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.363127 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.364604 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.365059 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.365202 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:54:21.368361 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:54:21.368367 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:54:21.368371 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:54:21.368413 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:54:21.371687 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:54:21.372663 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:54:21.372672 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:54:21.372676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:54:21.372822 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:54:21.373259 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:54:21.373267 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:54:21.373271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:54:21.373588 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:54:21.373861 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 02:54:21.374230 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 02:54:21.374515 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 02:54:21.379878 11726 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:54:21.388211 11726 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:54:21.414847 12230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:54:21.401916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36689 (local address 127.0.0.1:50962)
0620 02:54:21.402958 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0620 02:54:21.403019 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 02:54:21.403248 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:54:21.407028 (+ 3780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:54:21.407034 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:54:21.407063 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:54:21.407561 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:54:21.407573 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.409603 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.409612 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:54:21.412200 (+ 2588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:54:21.412214 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.413768 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.413775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:54:21.413993 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:54:21.414047 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 02:54:21.414202 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 02:54:21.414459 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":1754,"threads_started":1}
I20260620 02:54:21.414841 12231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:54:21.402204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50962 (local address 127.0.97.2:36689)
0620 02:54:21.406276 (+ 4072us) server_negotiation.cc:206] Beginning negotiation
0620 02:54:21.406285 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:54:21.406312 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:54:21.406452 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:54:21.406463 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:54:21.406797 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:54:21.407682 (+ 885us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:54:21.407876 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.409379 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.412391 (+ 3012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.413551 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.413727 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:54:21.413957 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 02:54:21.414249 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 02:54:21.414453 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3879,"thread_start_us":3762,"threads_started":1}
I20260620 02:54:21.415638 12234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:54:21.402683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40957 (local address 127.0.0.1:56250)
0620 02:54:21.406047 (+ 3364us) negotiation.cc:107] Waiting for socket to connect
0620 02:54:21.406066 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:54:21.406207 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:54:21.406906 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:54:21.406914 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:54:21.406946 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:54:21.407430 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:54:21.407444 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.409490 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.409498 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:54:21.411516 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:54:21.411529 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.414422 (+ 2893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.414428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:54:21.414729 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:54:21.414856 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 02:54:21.414952 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 02:54:21.415107 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3034,"mutex_wait_us":120,"thread_start_us":2658,"threads_started":1}
I20260620 02:54:21.415745 12233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:54:21.402458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38414 (local address 127.0.97.3:39709)
0620 02:54:21.405241 (+ 2783us) server_negotiation.cc:206] Beginning negotiation
0620 02:54:21.405255 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:54:21.405293 (+ 38us) server_negotiation.cc:407] Connection header received
0620 02:54:21.405393 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:54:21.405402 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:54:21.405698 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:54:21.406480 (+ 782us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:54:21.406801 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.408357 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.411666 (+ 3309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.413773 (+ 2107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.413930 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:54:21.414177 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 02:54:21.415067 (+ 890us) server_negotiation.cc:299] Negotiation successful
0620 02:54:21.415380 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2152,"thread_start_us":2108,"threads_started":1}
I20260620 02:54:21.416275 12235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:54:21.403235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56250 (local address 127.0.97.1:40957)
0620 02:54:21.404643 (+ 1408us) server_negotiation.cc:206] Beginning negotiation
0620 02:54:21.404653 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:54:21.406235 (+ 1582us) server_negotiation.cc:407] Connection header received
0620 02:54:21.406407 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:54:21.406418 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:54:21.406687 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:54:21.406898 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:54:21.407623 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.409343 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.411947 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.413854 (+ 1907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.413972 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:54:21.414185 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 02:54:21.415503 (+ 1318us) server_negotiation.cc:299] Negotiation successful
0620 02:54:21.415704 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1217,"thread_start_us":2248,"threads_started":1}
I20260620 02:54:21.416406 12232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:54:21.401915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39709 (local address 127.0.0.1:38414)
0620 02:54:21.403955 (+ 2040us) negotiation.cc:107] Waiting for socket to connect
0620 02:54:21.403973 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:54:21.404075 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:54:21.405968 (+ 1893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:54:21.405978 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:54:21.406033 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:54:21.406519 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:54:21.406533 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.408522 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.408529 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:54:21.411468 (+ 2939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:54:21.411484 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.414400 (+ 2916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.414409 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:54:21.414722 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:54:21.414863 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 02:54:21.414943 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 02:54:21.416037 (+ 1094us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1917,"mutex_wait_us":1319,"thread_start_us":1485,"threads_started":1}
I20260620 02:54:21.419994 12018 tablet_service.cc:1467] Processing CreateTablet for tablet dddc324066a14cacbda212f25765a3ef (DEFAULT_TABLE table=test-table [id=c4c182356c554991bf6640f77f88ceec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:54:21.420282 12152 tablet_service.cc:1467] Processing CreateTablet for tablet dddc324066a14cacbda212f25765a3ef (DEFAULT_TABLE table=test-table [id=c4c182356c554991bf6640f77f88ceec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:54:21.421072 11875 tablet_service.cc:1467] Processing CreateTablet for tablet dddc324066a14cacbda212f25765a3ef (DEFAULT_TABLE table=test-table [id=c4c182356c554991bf6640f77f88ceec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:54:21.421622 12152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dddc324066a14cacbda212f25765a3ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:54:21.421624 12018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dddc324066a14cacbda212f25765a3ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:54:21.422549 11875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dddc324066a14cacbda212f25765a3ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:54:21.434262 12236 tablet_bootstrap.cc:492] T dddc324066a14cacbda212f25765a3ef P 7e870a45402841deb26401373042cf0e: Bootstrap starting.
I20260620 02:54:21.438634 12238 tablet_bootstrap.cc:492] T dddc324066a14cacbda212f25765a3ef P 88ecaa37705c4e1a8be53e56c63231c7: Bootstrap starting.
I20260620 02:54:21.438958 12236 tablet_bootstrap.cc:654] T dddc324066a14cacbda212f25765a3ef P 7e870a45402841deb26401373042cf0e: Neither blocks nor log segments found. Creating new log.
I20260620 02:54:21.439486 12237 tablet_bootstrap.cc:492] T dddc324066a14cacbda212f25765a3ef P e9d5c598d2df4cb5bffba30363c51ff2: Bootstrap starting.
I20260620 02:54:21.440446 12236 log.cc:824] T dddc324066a14cacbda212f25765a3ef P 7e870a45402841deb26401373042cf0e: Log is configured to *not* fsync() on all Append() calls
I20260620 02:54:21.443593 12237 tablet_bootstrap.cc:654] T dddc324066a14cacbda212f25765a3ef P e9d5c598d2df4cb5bffba30363c51ff2: Neither blocks nor log segments found. Creating new log.
I20260620 02:54:21.443797 12238 tablet_bootstrap.cc:654] T dddc324066a14cacbda212f25765a3ef P 88ecaa37705c4e1a8be53e56c63231c7: Neither blocks nor log segments found. Creating new log.
I20260620 02:54:21.444923 12237 log.cc:824] T dddc324066a14cacbda212f25765a3ef P e9d5c598d2df4cb5bffba30363c51ff2: Log is configured to *not* fsync() on all Append() calls
I20260620 02:54:21.445075 12238 log.cc:824] T dddc324066a14cacbda212f25765a3ef P 88ecaa37705c4e1a8be53e56c63231c7: Log is configured to *not* fsync() on all Append() calls
I20260620 02:54:21.445431 12236 tablet_bootstrap.cc:492] T dddc324066a14cacbda212f25765a3ef P 7e870a45402841deb26401373042cf0e: No bootstrap required, opened a new log
I20260620 02:54:21.445957 12236 ts_tablet_manager.cc:1397] T dddc324066a14cacbda212f25765a3ef P 7e870a45402841deb26401373042cf0e: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260620 02:54:21.448385 12237 tablet_bootstrap.cc:492] T dddc324066a14cacbda212f25765a3ef P e9d5c598d2df4cb5bffba30363c51ff2: No bootstrap required, opened a new log
I20260620 02:54:21.448683 12237 ts_tablet_manager.cc:1397] T dddc324066a14cacbda212f25765a3ef P e9d5c598d2df4cb5bffba30363c51ff2: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.008s
I20260620 02:54:21.450155 12238 tablet_bootstrap.cc:492] T dddc324066a14cacbda212f25765a3ef P 88ecaa37705c4e1a8be53e56c63231c7: No bootstrap required, opened a new log
I20260620 02:54:21.450497 12238 ts_tablet_manager.cc:1397] T dddc324066a14cacbda212f25765a3ef P 88ecaa37705c4e1a8be53e56c63231c7: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260620 02:54:21.458782 12238 raft_consensus.cc:348] T dddc324066a14cacbda212f25765a3ef P 88ecaa37705c4e1a8be53e56c63231c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ecaa37705c4e1a8be53e56c63231c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39709 } } peers { permanent_uuid: "e9d5c598d2df4cb5bffba30363c51ff2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36689 } } peers { permanent_uuid: "7e870a45402841deb26401373042cf0e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40957 } }
I20260620 02:54:21.459267 12238 raft_consensus.cc:374] T dddc324066a14cacbda212f25765a3ef P 88ecaa37705c4e1a8be53e56c63231c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:54:21.458874 12236 raft_consensus.cc:348] T dddc324066a14cacbda212f25765a3ef P 7e870a45402841deb26401373042cf0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ecaa37705c4e1a8be53e56c63231c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39709 } } peers { permanent_uuid: "e9d5c598d2df4cb5bffba30363c51ff2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36689 } } peers { permanent_uuid: "7e870a45402841deb26401373042cf0e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40957 } }
I20260620 02:54:21.459434 12236 raft_consensus.cc:374] T dddc324066a14cacbda212f25765a3ef P 7e870a45402841deb26401373042cf0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:54:21.459424 12238 raft_consensus.cc:729] T dddc324066a14cacbda212f25765a3ef P 88ecaa37705c4e1a8be53e56c63231c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88ecaa37705c4e1a8be53e56c63231c7, State: Initialized, Role: FOLLOWER
I20260620 02:54:21.459569 12236 raft_consensus.cc:729] T dddc324066a14cacbda212f25765a3ef P 7e870a45402841deb26401373042cf0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e870a45402841deb26401373042cf0e, State: Initialized, Role: FOLLOWER
I20260620 02:54:21.460265 12238 consensus_queue.cc:260] T dddc324066a14cacbda212f25765a3ef P 88ecaa37705c4e1a8be53e56c63231c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ecaa37705c4e1a8be53e56c63231c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39709 } } peers { permanent_uuid: "e9d5c598d2df4cb5bffba30363c51ff2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36689 } } peers { permanent_uuid: "7e870a45402841deb26401373042cf0e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40957 } }
I20260620 02:54:21.460299 12236 consensus_queue.cc:260] T dddc324066a14cacbda212f25765a3ef P 7e870a45402841deb26401373042cf0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ecaa37705c4e1a8be53e56c63231c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39709 } } peers { permanent_uuid: "e9d5c598d2df4cb5bffba30363c51ff2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36689 } } peers { permanent_uuid: "7e870a45402841deb26401373042cf0e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40957 } }
I20260620 02:54:21.461622 12237 raft_consensus.cc:348] T dddc324066a14cacbda212f25765a3ef P e9d5c598d2df4cb5bffba30363c51ff2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ecaa37705c4e1a8be53e56c63231c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39709 } } peers { permanent_uuid: "e9d5c598d2df4cb5bffba30363c51ff2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36689 } } peers { permanent_uuid: "7e870a45402841deb26401373042cf0e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40957 } }
I20260620 02:54:21.462126 12237 raft_consensus.cc:374] T dddc324066a14cacbda212f25765a3ef P e9d5c598d2df4cb5bffba30363c51ff2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:54:21.462270 12237 raft_consensus.cc:729] T dddc324066a14cacbda212f25765a3ef P e9d5c598d2df4cb5bffba30363c51ff2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d5c598d2df4cb5bffba30363c51ff2, State: Initialized, Role: FOLLOWER
I20260620 02:54:21.462498 12236 ts_tablet_manager.cc:1428] T dddc324066a14cacbda212f25765a3ef P 7e870a45402841deb26401373042cf0e: Time spent starting tablet: real 0.016s user 0.007s sys 0.009s
I20260620 02:54:21.463075 12237 consensus_queue.cc:260] T dddc324066a14cacbda212f25765a3ef P e9d5c598d2df4cb5bffba30363c51ff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ecaa37705c4e1a8be53e56c63231c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39709 } } peers { permanent_uuid: "e9d5c598d2df4cb5bffba30363c51ff2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36689 } } peers { permanent_uuid: "7e870a45402841deb26401373042cf0e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40957 } }
I20260620 02:54:21.463562 12217 heartbeater.cc:503] Master 127.0.97.62:41507 was elected leader, sending a full tablet report...
I20260620 02:54:21.464282 12238 ts_tablet_manager.cc:1428] T dddc324066a14cacbda212f25765a3ef P 88ecaa37705c4e1a8be53e56c63231c7: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260620 02:54:21.465376 12083 heartbeater.cc:503] Master 127.0.97.62:41507 was elected leader, sending a full tablet report...
I20260620 02:54:21.466051 12237 ts_tablet_manager.cc:1428] T dddc324066a14cacbda212f25765a3ef P e9d5c598d2df4cb5bffba30363c51ff2: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
W20260620 02:54:21.494705 12084 tablet.cc:2369] T dddc324066a14cacbda212f25765a3ef P e9d5c598d2df4cb5bffba30363c51ff2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:54:21.535892 12242 raft_consensus.cc:482] T dddc324066a14cacbda212f25765a3ef P 7e870a45402841deb26401373042cf0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:54:21.536182 12242 raft_consensus.cc:504] T dddc324066a14cacbda212f25765a3ef P 7e870a45402841deb26401373042cf0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ecaa37705c4e1a8be53e56c63231c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39709 } } peers { permanent_uuid: "e9d5c598d2df4cb5bffba30363c51ff2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36689 } } peers { permanent_uuid: "7e870a45402841deb26401373042cf0e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40957 } }
I20260620 02:54:21.537869 12242 leader_election.cc:288] T dddc324066a14cacbda212f25765a3ef P 7e870a45402841deb26401373042cf0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88ecaa37705c4e1a8be53e56c63231c7 (127.0.97.3:39709), e9d5c598d2df4cb5bffba30363c51ff2 (127.0.97.2:36689)
I20260620 02:54:21.548111 12231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:54:21.538314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58967 (local address 127.0.97.2:36689)
0620 02:54:21.538557 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 02:54:21.538564 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:54:21.539783 (+ 1219us) server_negotiation.cc:407] Connection header received
0620 02:54:21.540020 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:54:21.540025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:54:21.540114 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:54:21.540227 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:54:21.541116 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.542217 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.545524 (+ 3307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.546849 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.547008 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:54:21.547125 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 02:54:21.547671 (+ 546us) server_negotiation.cc:299] Negotiation successful
0620 02:54:21.547807 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260620 02:54:21.548310 12246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:54:21.538282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36689 (local address 127.0.97.1:58967)
0620 02:54:21.539639 (+ 1357us) negotiation.cc:107] Waiting for socket to connect
0620 02:54:21.539654 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:54:21.539846 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:54:21.540314 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:54:21.540319 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:54:21.540369 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:54:21.540945 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:54:21.540959 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.542406 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.542412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:54:21.545303 (+ 2891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:54:21.545321 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.547203 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.547211 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:54:21.547503 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:54:21.547528 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 02:54:21.547618 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 02:54:21.547842 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1262,"mutex_wait_us":382,"spinlock_wait_cycles":54144,"thread_start_us":1001,"threads_started":1}
I20260620 02:54:21.548079 12233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:54:21.538314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42819 (local address 127.0.97.3:39709)
0620 02:54:21.538557 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 02:54:21.538564 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:54:21.539778 (+ 1214us) server_negotiation.cc:407] Connection header received
0620 02:54:21.540020 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:54:21.540025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:54:21.540109 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:54:21.540257 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:54:21.541116 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.542216 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.545551 (+ 3335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.546862 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.547003 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:54:21.547159 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 02:54:21.547671 (+ 512us) server_negotiation.cc:299] Negotiation successful
0620 02:54:21.547807 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 02:54:21.548643 12245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:54:21.538244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39709 (local address 127.0.97.1:42819)
0620 02:54:21.539607 (+ 1363us) negotiation.cc:107] Waiting for socket to connect
0620 02:54:21.539633 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:54:21.539807 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:54:21.540314 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:54:21.540319 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:54:21.540367 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:54:21.540951 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:54:21.540959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.542409 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.542413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:54:21.545303 (+ 2890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:54:21.545321 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.547221 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.547226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:54:21.547441 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:54:21.547508 (+ 67us) client_negotiation.cc:769] Sending connection context
0620 02:54:21.547618 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 02:54:21.547842 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1197,"mutex_wait_us":358,"spinlock_wait_cycles":12288,"thread_start_us":990,"threads_started":1}
I20260620 02:54:21.548631 12244 raft_consensus.cc:482] T dddc324066a14cacbda212f25765a3ef P e9d5c598d2df4cb5bffba30363c51ff2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:54:21.548936 12244 raft_consensus.cc:504] T dddc324066a14cacbda212f25765a3ef P e9d5c598d2df4cb5bffba30363c51ff2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ecaa37705c4e1a8be53e56c63231c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39709 } } peers { permanent_uuid: "e9d5c598d2df4cb5bffba30363c51ff2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36689 } } peers { permanent_uuid: "7e870a45402841deb26401373042cf0e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40957 } }
I20260620 02:54:21.549686 12172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dddc324066a14cacbda212f25765a3ef" candidate_uuid: "7e870a45402841deb26401373042cf0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88ecaa37705c4e1a8be53e56c63231c7" is_pre_election: true
I20260620 02:54:21.549743 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dddc324066a14cacbda212f25765a3ef" candidate_uuid: "7e870a45402841deb26401373042cf0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d5c598d2df4cb5bffba30363c51ff2" is_pre_election: true
I20260620 02:54:21.550179 12172 raft_consensus.cc:2454] T dddc324066a14cacbda212f25765a3ef P 88ecaa37705c4e1a8be53e56c63231c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e870a45402841deb26401373042cf0e in term 0.
I20260620 02:54:21.550285 12038 raft_consensus.cc:2454] T dddc324066a14cacbda212f25765a3ef P e9d5c598d2df4cb5bffba30363c51ff2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e870a45402841deb26401373042cf0e in term 0.
I20260620 02:54:21.550923 11830 leader_election.cc:302] T dddc324066a14cacbda212f25765a3ef P 7e870a45402841deb26401373042cf0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e870a45402841deb26401373042cf0e, 88ecaa37705c4e1a8be53e56c63231c7; no voters:
I20260620 02:54:21.551335 12242 raft_consensus.cc:2781] T dddc324066a14cacbda212f25765a3ef P 7e870a45402841deb26401373042cf0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:54:21.551553 12242 raft_consensus.cc:482] T dddc324066a14cacbda212f25765a3ef P 7e870a45402841deb26401373042cf0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:54:21.551690 12242 raft_consensus.cc:3037] T dddc324066a14cacbda212f25765a3ef P 7e870a45402841deb26401373042cf0e [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:54:21.553885 12244 leader_election.cc:288] T dddc324066a14cacbda212f25765a3ef P e9d5c598d2df4cb5bffba30363c51ff2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88ecaa37705c4e1a8be53e56c63231c7 (127.0.97.3:39709), 7e870a45402841deb26401373042cf0e (127.0.97.1:40957)
I20260620 02:54:21.555408 12242 raft_consensus.cc:504] T dddc324066a14cacbda212f25765a3ef P 7e870a45402841deb26401373042cf0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ecaa37705c4e1a8be53e56c63231c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39709 } } peers { permanent_uuid: "e9d5c598d2df4cb5bffba30363c51ff2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36689 } } peers { permanent_uuid: "7e870a45402841deb26401373042cf0e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40957 } }
I20260620 02:54:21.556274 12242 leader_election.cc:288] T dddc324066a14cacbda212f25765a3ef P 7e870a45402841deb26401373042cf0e [CANDIDATE]: Term 1 election: Requested vote from peers 88ecaa37705c4e1a8be53e56c63231c7 (127.0.97.3:39709), e9d5c598d2df4cb5bffba30363c51ff2 (127.0.97.2:36689)
I20260620 02:54:21.556747 12172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dddc324066a14cacbda212f25765a3ef" candidate_uuid: "7e870a45402841deb26401373042cf0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88ecaa37705c4e1a8be53e56c63231c7"
I20260620 02:54:21.556833 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dddc324066a14cacbda212f25765a3ef" candidate_uuid: "7e870a45402841deb26401373042cf0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d5c598d2df4cb5bffba30363c51ff2"
I20260620 02:54:21.556975 12172 raft_consensus.cc:3037] T dddc324066a14cacbda212f25765a3ef P 88ecaa37705c4e1a8be53e56c63231c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:54:21.557073 12038 raft_consensus.cc:3037] T dddc324066a14cacbda212f25765a3ef P e9d5c598d2df4cb5bffba30363c51ff2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:54:21.560496 12038 raft_consensus.cc:2454] T dddc324066a14cacbda212f25765a3ef P e9d5c598d2df4cb5bffba30363c51ff2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e870a45402841deb26401373042cf0e in term 1.
I20260620 02:54:21.560758 12172 raft_consensus.cc:2454] T dddc324066a14cacbda212f25765a3ef P 88ecaa37705c4e1a8be53e56c63231c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e870a45402841deb26401373042cf0e in term 1.
I20260620 02:54:21.561026 11831 leader_election.cc:302] T dddc324066a14cacbda212f25765a3ef P 7e870a45402841deb26401373042cf0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e870a45402841deb26401373042cf0e, e9d5c598d2df4cb5bffba30363c51ff2; no voters:
I20260620 02:54:21.561522 12242 raft_consensus.cc:2781] T dddc324066a14cacbda212f25765a3ef P 7e870a45402841deb26401373042cf0e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:54:21.561743 12233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:54:21.552715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57911 (local address 127.0.97.3:39709)
0620 02:54:21.553061 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0620 02:54:21.553067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:54:21.554020 (+ 953us) server_negotiation.cc:407] Connection header received
0620 02:54:21.554255 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:54:21.554264 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:54:21.554316 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:54:21.554452 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:54:21.555071 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.555847 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.558522 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.560465 (+ 1943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.560602 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:54:21.560719 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 02:54:21.561313 (+ 594us) server_negotiation.cc:299] Negotiation successful
0620 02:54:21.561448 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244}
I20260620 02:54:21.563060 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:54:21.551515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39709 (local address 127.0.97.2:57911)
0620 02:54:21.553929 (+ 2414us) negotiation.cc:107] Waiting for socket to connect
0620 02:54:21.553946 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:54:21.554117 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:54:21.554480 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:54:21.554484 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:54:21.554513 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:54:21.554937 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:54:21.554946 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.556048 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.556052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:54:21.558310 (+ 2258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:54:21.558330 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.560890 (+ 2560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.560896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:54:21.561121 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:54:21.561161 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 02:54:21.562010 (+ 849us) client_negotiation.cc:240] Negotiation successful
0620 02:54:21.562211 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2287,"mutex_wait_us":21,"thread_start_us":1708,"threads_started":1}
I20260620 02:54:21.563488 12235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:54:21.552418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40579 (local address 127.0.97.1:40957)
0620 02:54:21.552929 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0620 02:54:21.552937 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:54:21.553664 (+ 727us) server_negotiation.cc:407] Connection header received
0620 02:54:21.553943 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:54:21.553948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:54:21.554018 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:54:21.554090 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:54:21.554938 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.556060 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.561033 (+ 4973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.562239 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.562316 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:54:21.562441 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 02:54:21.563108 (+ 667us) server_negotiation.cc:299] Negotiation successful
0620 02:54:21.563238 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362}
I20260620 02:54:21.563741 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:54:21.551481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40957 (local address 127.0.97.2:40579)
0620 02:54:21.553548 (+ 2067us) negotiation.cc:107] Waiting for socket to connect
0620 02:54:21.553569 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:54:21.553785 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:54:21.554291 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:54:21.554297 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:54:21.554321 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:54:21.554767 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:54:21.554779 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.556688 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.556694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:54:21.560824 (+ 4130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:54:21.560840 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.562785 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.562791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:54:21.562922 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:54:21.562948 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 02:54:21.563085 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 02:54:21.563263 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1931,"mutex_wait_us":23,"thread_start_us":2140,"threads_started":1}
I20260620 02:54:21.564397 12242 raft_consensus.cc:686] T dddc324066a14cacbda212f25765a3ef P 7e870a45402841deb26401373042cf0e [term 1 LEADER]: Becoming Leader. State: Replica: 7e870a45402841deb26401373042cf0e, State: Running, Role: LEADER
I20260620 02:54:21.564668 12172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dddc324066a14cacbda212f25765a3ef" candidate_uuid: "e9d5c598d2df4cb5bffba30363c51ff2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88ecaa37705c4e1a8be53e56c63231c7" is_pre_election: true
I20260620 02:54:21.565056 12172 raft_consensus.cc:2379] T dddc324066a14cacbda212f25765a3ef P 88ecaa37705c4e1a8be53e56c63231c7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e9d5c598d2df4cb5bffba30363c51ff2 in current term 1: Already voted for candidate 7e870a45402841deb26401373042cf0e in this term.
I20260620 02:54:21.564920 11895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dddc324066a14cacbda212f25765a3ef" candidate_uuid: "e9d5c598d2df4cb5bffba30363c51ff2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e870a45402841deb26401373042cf0e" is_pre_election: true
I20260620 02:54:21.565052 12242 consensus_queue.cc:237] T dddc324066a14cacbda212f25765a3ef P 7e870a45402841deb26401373042cf0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ecaa37705c4e1a8be53e56c63231c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39709 } } peers { permanent_uuid: "e9d5c598d2df4cb5bffba30363c51ff2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36689 } } peers { permanent_uuid: "7e870a45402841deb26401373042cf0e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40957 } }
I20260620 02:54:21.566128 11972 leader_election.cc:302] T dddc324066a14cacbda212f25765a3ef P e9d5c598d2df4cb5bffba30363c51ff2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d5c598d2df4cb5bffba30363c51ff2; no voters: 7e870a45402841deb26401373042cf0e, 88ecaa37705c4e1a8be53e56c63231c7
I20260620 02:54:21.566630 12244 raft_consensus.cc:2726] T dddc324066a14cacbda212f25765a3ef P e9d5c598d2df4cb5bffba30363c51ff2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260620 02:54:21.568428 12218 tablet.cc:2369] T dddc324066a14cacbda212f25765a3ef P 88ecaa37705c4e1a8be53e56c63231c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:54:21.570292 11726 catalog_manager.cc:5495] T dddc324066a14cacbda212f25765a3ef P 7e870a45402841deb26401373042cf0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e870a45402841deb26401373042cf0e (127.0.97.1). New cstate: current_term: 1 leader_uuid: "7e870a45402841deb26401373042cf0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ecaa37705c4e1a8be53e56c63231c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d5c598d2df4cb5bffba30363c51ff2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e870a45402841deb26401373042cf0e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40957 } health_report { overall_health: HEALTHY } } }
I20260620 02:54:21.610903 12235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:54:21.605112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56258 (local address 127.0.97.1:40957)
0620 02:54:21.605347 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 02:54:21.605353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:54:21.605455 (+ 102us) server_negotiation.cc:407] Connection header received
0620 02:54:21.605745 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:54:21.605752 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:54:21.605856 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:54:21.606013 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:54:21.606475 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.607243 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.608689 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.609106 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.609241 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:54:21.609770 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:54:21.610280 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:54:21.610480 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 02:54:21.610529 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 02:54:21.610607 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 02:54:21.619316 12038 raft_consensus.cc:1261] T dddc324066a14cacbda212f25765a3ef P e9d5c598d2df4cb5bffba30363c51ff2 [term 1 FOLLOWER]: Refusing update from remote peer 7e870a45402841deb26401373042cf0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:54:21.619428 12172 raft_consensus.cc:1261] T dddc324066a14cacbda212f25765a3ef P 88ecaa37705c4e1a8be53e56c63231c7 [term 1 FOLLOWER]: Refusing update from remote peer 7e870a45402841deb26401373042cf0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:54:21.620298 12249 consensus_queue.cc:1035] T dddc324066a14cacbda212f25765a3ef P 7e870a45402841deb26401373042cf0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d5c598d2df4cb5bffba30363c51ff2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:54:21.620668 12242 consensus_queue.cc:1035] T dddc324066a14cacbda212f25765a3ef P 7e870a45402841deb26401373042cf0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88ecaa37705c4e1a8be53e56c63231c7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:54:21.635705 12254 mvcc.cc:205] Tried to move back new op lower bound from 7298760956383358976 to 7298760956175069184. Current Snapshot: MvccSnapshot[applied={T|T < 7298760956383358976}]
I20260620 02:54:21.648468 11727 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:54:21.648837 11727 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:54:21.652714 11727 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b7211d166be4d64b99566cf68f7e40b: Sending DeleteTablet for 3 replicas of tablet dddc324066a14cacbda212f25765a3ef
I20260620 02:54:21.653951 12152 tablet_service.cc:1514] Processing DeleteTablet for tablet dddc324066a14cacbda212f25765a3ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:54:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38414
I20260620 02:54:21.653968 11875 tablet_service.cc:1514] Processing DeleteTablet for tablet dddc324066a14cacbda212f25765a3ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:54:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56250
I20260620 02:54:21.654018 12018 tablet_service.cc:1514] Processing DeleteTablet for tablet dddc324066a14cacbda212f25765a3ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:54:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50962
I20260620 02:54:21.657780 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 11807
I20260620 02:54:21.669234 12263 tablet_replica.cc:330] T dddc324066a14cacbda212f25765a3ef P 88ecaa37705c4e1a8be53e56c63231c7: stopping tablet replica
I20260620 02:54:21.669958 12263 raft_consensus.cc:2229] T dddc324066a14cacbda212f25765a3ef P 88ecaa37705c4e1a8be53e56c63231c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:54:21.670696 12263 raft_consensus.cc:2258] T dddc324066a14cacbda212f25765a3ef P 88ecaa37705c4e1a8be53e56c63231c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:54:21.673313 12263 ts_tablet_manager.cc:1905] T dddc324066a14cacbda212f25765a3ef P 88ecaa37705c4e1a8be53e56c63231c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:54:21.681205 12263 ts_tablet_manager.cc:1918] T dddc324066a14cacbda212f25765a3ef P 88ecaa37705c4e1a8be53e56c63231c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:54:21.681466 12263 log.cc:1196] T dddc324066a14cacbda212f25765a3ef P 88ecaa37705c4e1a8be53e56c63231c7: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-2/wal/wals/dddc324066a14cacbda212f25765a3ef
I20260620 02:54:21.682072 12263 ts_tablet_manager.cc:1939] T dddc324066a14cacbda212f25765a3ef P 88ecaa37705c4e1a8be53e56c63231c7: Deleting consensus metadata
I20260620 02:54:21.684276 11713 catalog_manager.cc:4841] TS 88ecaa37705c4e1a8be53e56c63231c7 (127.0.97.3:39709): tablet dddc324066a14cacbda212f25765a3ef (table test-table [id=c4c182356c554991bf6640f77f88ceec]) successfully deleted
I20260620 02:54:21.688894 12261 tablet_replica.cc:330] T dddc324066a14cacbda212f25765a3ef P 7e870a45402841deb26401373042cf0e: stopping tablet replica
I20260620 02:54:21.689725 12261 raft_consensus.cc:2229] T dddc324066a14cacbda212f25765a3ef P 7e870a45402841deb26401373042cf0e [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:54:21.690734 12261 raft_consensus.cc:2258] T dddc324066a14cacbda212f25765a3ef P 7e870a45402841deb26401373042cf0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:54:21.691728 12262 tablet_replica.cc:330] T dddc324066a14cacbda212f25765a3ef P e9d5c598d2df4cb5bffba30363c51ff2: stopping tablet replica
I20260620 02:54:21.692274 12262 raft_consensus.cc:2229] T dddc324066a14cacbda212f25765a3ef P e9d5c598d2df4cb5bffba30363c51ff2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:54:21.692865 12262 raft_consensus.cc:2258] T dddc324066a14cacbda212f25765a3ef P e9d5c598d2df4cb5bffba30363c51ff2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:54:21.693640 12261 ts_tablet_manager.cc:1905] T dddc324066a14cacbda212f25765a3ef P 7e870a45402841deb26401373042cf0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:54:21.695158 12262 ts_tablet_manager.cc:1905] T dddc324066a14cacbda212f25765a3ef P e9d5c598d2df4cb5bffba30363c51ff2: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 02:54:21 dist-test-slave-8p8c krb5kdc[11251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:54:21.701872 12261 ts_tablet_manager.cc:1918] T dddc324066a14cacbda212f25765a3ef P 7e870a45402841deb26401373042cf0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:54:21.702101 12261 log.cc:1196] T dddc324066a14cacbda212f25765a3ef P 7e870a45402841deb26401373042cf0e: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-0/wal/wals/dddc324066a14cacbda212f25765a3ef
I20260620 02:54:21.702488 12262 ts_tablet_manager.cc:1918] T dddc324066a14cacbda212f25765a3ef P e9d5c598d2df4cb5bffba30363c51ff2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:54:21.702705 12262 log.cc:1196] T dddc324066a14cacbda212f25765a3ef P e9d5c598d2df4cb5bffba30363c51ff2: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781923914418765-388-0/minicluster-data/ts-1/wal/wals/dddc324066a14cacbda212f25765a3ef
I20260620 02:54:21.702773 12261 ts_tablet_manager.cc:1939] T dddc324066a14cacbda212f25765a3ef P 7e870a45402841deb26401373042cf0e: Deleting consensus metadata
I20260620 02:54:21.703266 12262 ts_tablet_manager.cc:1939] T dddc324066a14cacbda212f25765a3ef P e9d5c598d2df4cb5bffba30363c51ff2: Deleting consensus metadata
I20260620 02:54:21.704686 11712 catalog_manager.cc:4841] TS 7e870a45402841deb26401373042cf0e (127.0.97.1:40957): tablet dddc324066a14cacbda212f25765a3ef (table test-table [id=c4c182356c554991bf6640f77f88ceec]) successfully deleted
I20260620 02:54:21.705085 11714 catalog_manager.cc:4841] TS e9d5c598d2df4cb5bffba30363c51ff2 (127.0.97.2:36689): tablet dddc324066a14cacbda212f25765a3ef (table test-table [id=c4c182356c554991bf6640f77f88ceec]) successfully deleted
I20260620 02:54:21.707093 12235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:54:21.659527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56274 (local address 127.0.97.1:40957)
0620 02:54:21.659750 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 02:54:21.659779 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0620 02:54:21.683724 (+ 23945us) server_negotiation.cc:407] Connection header received
0620 02:54:21.684385 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:54:21.684395 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:54:21.684552 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:54:21.684666 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:54:21.687538 (+ 2872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.688860 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.691109 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.691939 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.692055 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:54:21.698287 (+ 6232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:54:21.698302 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:54:21.698322 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:54:21.698442 (+ 120us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:54:21.703434 (+ 4992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:54:21.704404 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:54:21.704414 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:54:21.704417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:54:21.704573 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:54:21.705227 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:54:21.705232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:54:21.705235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:54:21.705805 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:54:21.706101 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 02:54:21.706472 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 02:54:21.706737 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 02:54:21.707757 11935 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:54:21.809176 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 11807
I20260620 02:54:21.824357 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 11946
Jun 20 02:54:21 dist-test-slave-8p8c krb5kdc[11251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:54:21.839443 12231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:54:21.825100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50978 (local address 127.0.97.2:36689)
0620 02:54:21.825326 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 02:54:21.825330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:54:21.825347 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:54:21.825479 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:54:21.825483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:54:21.825545 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:54:21.825654 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:54:21.827117 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.827845 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.829114 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.829508 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.829608 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:54:21.833816 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:54:21.833830 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:54:21.833846 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:54:21.833887 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:54:21.836836 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:54:21.837491 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:54:21.837499 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:54:21.837501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:54:21.837638 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:54:21.837952 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:54:21.837955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:54:21.837957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:54:21.838325 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:54:21.838618 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 02:54:21.838884 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 02:54:21.839081 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 02:54:21.840191 12078 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:54:21.934561 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 11946
I20260620 02:54:21.953915 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 12089
Jun 20 02:54:21 dist-test-slave-8p8c krb5kdc[11251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:54:21.970923 12233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:54:21.954902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38424 (local address 127.0.97.3:39709)
0620 02:54:21.955139 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 02:54:21.955146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:54:21.955238 (+ 92us) server_negotiation.cc:407] Connection header received
0620 02:54:21.955465 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:54:21.955469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:54:21.955533 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:54:21.955649 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:54:21.957276 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.957983 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.959418 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:54:21.959939 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:54:21.960080 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:54:21.964990 (+ 4910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:54:21.965009 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:54:21.965031 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:54:21.965081 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:54:21.968080 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:54:21.968851 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:54:21.968859 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:54:21.968861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:54:21.969012 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:54:21.969443 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:54:21.969447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:54:21.969450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:54:21.969816 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:54:21.970063 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 02:54:21.970513 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 02:54:21.970688 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260620 02:54:21.971580 12212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:54:22.082350 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 12089
I20260620 02:54:22.098663 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 11695
I20260620 02:54:22.099792 11757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:54:22.215377 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 11695
I20260620 02:54:22.237301 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 02:54:22.573089 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 02:54:22.573354 388 mini_ranger.cc:66] Stopping Ranger...
I20260620 02:54:22.919799 388 mini_ranger.cc:68] Stopped Ranger
2026-06-20 02:54:22.920 UTC [11281] LOG: received smart shutdown request
2026-06-20 02:54:22.924 UTC [11281] LOG: background worker "logical replication launcher" (PID 11290) exited with exit code 1
2026-06-20 02:54:22.924 UTC [11285] LOG: shutting down
2026-06-20 02:54:23.124 UTC [11281] LOG: database system is shut down
2026-06-20T02:54:23Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50350 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:54:23 dist-test-slave-8p8c krb5kdc[12277](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:54:23 dist-test-slave-8p8c krb5kdc[12277](info): set up 2 sockets
Jun 20 02:54:23 dist-test-slave-8p8c krb5kdc[12277](info): commencing operation
krb5kdc: starting...
W20260620 02:54:24.374703 388 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.001s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:54:24 dist-test-slave-8p8c krb5kdc[12277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:54:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:54:24Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260620 02:54:24.705948 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskgxGzum/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/postgres -l logfile start
2026-06-20 02:54:27.597 UTC [12307] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 02:54:27.597 UTC [12307] LOG: listening on IPv4 address "127.0.97.20", port 38643
2026-06-20 02:54:27.601 UTC [12307] LOG: listening on Unix socket "/tmp/.s.PGSQL.38643"
2026-06-20 02:54:27.625 UTC [12310] LOG: database system was shut down at 2026-06-20 02:54:25 UTC
2026-06-20 02:54:27.630 UTC [12307] LOG: database system is ready to accept connections
I20260620 02:54:28.638339 388 mini_postgres.cc:96] Postgres bound to 38643
2026-06-20 02:54:28.656 UTC [12318] FATAL: database "slave" does not exist
127.0.97.20:38643 - accepting connections
I20260620 02:54:28.657295 388 mini_ranger.cc:161] Starting Ranger...
I20260620 02:54:28.684958 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 02:54:29.322266 388 mini_ranger.cc:87] Created ranger Postgres database
I20260620 02:54:29.322402 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ranger-admin
2026-06-20 02:54:29,626 [I] DB FLAVOR :POSTGRES
2026-06-20 02:54:29,626 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 02:54:29,626 [I] Checking connection..
2026-06-20 02:54:29,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 02:54:29,902 [I] Checking connection passed.
2026-06-20 02:54:29,903 [I] --------- Verifying version history table ---------
2026-06-20 02:54:29,903 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 02:54:30,167 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 02:54:30,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 02:54:30,419 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 02:54:30,419 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 02:54:30,420 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38643/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 02:54:30.671 UTC [12382] WARNING: there is no transaction in progress
2026-06-20 02:54:30,677 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 02:54:30,678 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 02:54:30,949 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 02:54:30,950 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 02:54:30,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 02:54:31,206 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 02:54:31,474 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8p8c.c.gcp-upstream.internal','N') ;"
2026-06-20 02:54:31,712 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 02:54:31,967 [I] Table x_portal_user does not exist in database ranger
2026-06-20 02:54:31,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 02:54:32,241 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 02:54:32,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 02:54:32,488 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 02:54:32,721 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 02:54:32,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38643/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 02:54:33.748 UTC [12490] WARNING: there is no transaction in progress
2026-06-20 02:54:33.769 UTC [12490] WARNING: there is no transaction in progress
2026-06-20 02:54:33.788 UTC [12490] WARNING: there is no transaction in progress
2026-06-20 02:54:33.807 UTC [12490] WARNING: there is no transaction in progress
2026-06-20 02:54:33.824 UTC [12490] WARNING: there is no transaction in progress
2026-06-20 02:54:33.845 UTC [12490] WARNING: there is no transaction in progress
2026-06-20 02:54:33.905 UTC [12490] WARNING: there is no transaction in progress
2026-06-20 02:54:33.916 UTC [12490] WARNING: there is no transaction in progress
2026-06-20 02:54:33.935 UTC [12490] WARNING: there is no transaction in progress
2026-06-20 02:54:33.946 UTC [12490] WARNING: there is no transaction in progress
2026-06-20 02:54:33.958 UTC [12490] WARNING: there is no transaction in progress
2026-06-20 02:54:33.976 UTC [12490] WARNING: there is no transaction in progress
2026-06-20 02:54:33.988 UTC [12490] WARNING: there is no transaction in progress
2026-06-20 02:54:33.999 UTC [12490] WARNING: there is no transaction in progress
2026-06-20 02:54:34.009 UTC [12490] WARNING: there is no transaction in progress
2026-06-20 02:54:34.018 UTC [12490] WARNING: there is no transaction in progress
2026-06-20 02:54:34,751 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 02:54:34,752 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 02:54:35,025 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 02:54:35,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 02:54:35,308 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 02:54:35,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 02:54:35,581 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 02:54:35,582 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 02:54:35,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 02:54:36,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8p8c.c.gcp-upstream.internal';"
2026-06-20 02:54:36,309 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 02:54:36,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 02:54:36,561 [I] DB_PATCHES have already been applied
I20260620 02:54:36.567409 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskgxGzum/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 02:54:36.567553 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260620 02:54:36.581846 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:42549
Jun 20, 2026 2:54:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:54:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:54:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 2:54:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 2:54:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 2:54:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 2:54:37 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 02:54:37.724336 388 mini_ranger.cc:160] Time spent starting Ranger: real 9.067s user 0.001s sys 0.037s
Jun 20, 2026 2:54:58 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 2:54:59 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 2:54:59 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 2:54:59 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 2:54:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 2:54:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 2:54:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 2:54:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 2:54:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 2:54:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 2:54:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 2:54:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 2:54:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 2:54:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 2:54:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 2:54:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 2:54:59 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 2:54:59 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 02:55:00.571453 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 02:55:00 dist-test-slave-8p8c krb5kdc[12277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924100, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 02:55:00.761394 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 02:55:00.782465 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 02:55:02.001324 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 02:55:02.001425 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ranger-kms
I20260620 02:55:02.001461 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:38643
2026-06-20 02:55:02,365 [I] DB FLAVOR :POSTGRES
2026-06-20 02:55:02,365 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 02:55:02,365 [I] Checking connection
2026-06-20 02:55:02,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38643/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 02:55:02,635 [I] connection success
2026-06-20 02:55:02,635 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 02:55:02,636 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 02:55:02,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38643/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 02:55:02,877 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 02:55:02,877 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 02:55:02,877 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 02:55:02,877 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38643/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 02:55:03,150 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 02:55:03.674341 388 mini_ranger_kms.cc:310] Created kms service
I20260620 02:55:03.875209 388 mini_ranger_kms.cc:326] Created kudu user
I20260620 02:55:03.920706 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 02:55:03 dist-test-slave-8p8c krb5kdc[12277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924100, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:55:04.320338 388 mini_ranger_kms.cc:384] Added ranger policy
I20260620 02:55:04.320675 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskgxGzum/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskgxGzum/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 02:55:04.320734 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260620 02:55:04.341048 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12671
I20260620 02:55:04.341214 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:45969
Jun 20, 2026 2:55:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:55:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:55:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 2:55:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 2:55:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 2:55:05 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 02:55:06 dist-test-slave-8p8c krb5kdc[12277](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781924106, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:55:06.668911 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.908s user 0.090s sys 0.291s
I20260620 02:55:06.669106 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 02:55:06.669303 388 mini_ranger_kms.cc:401] 127.0.97.20:45969/kms/v1/keys
Jun 20 02:55:07 dist-test-slave-8p8c krb5kdc[12277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924106, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 02:55:07 dist-test-slave-8p8c krb5kdc[12277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924106, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 02:55:09 dist-test-slave-8p8c krb5kdc[12277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:55:09.735848 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33553
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33553
--ranger_config_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45969/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:55:10.072047 12721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:55:10.072379 12721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:55:10.072536 12721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:55:10.079746 12721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:55:10.079890 12721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:55:10.080000 12721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:55:10.080101 12721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:55:10.080183 12721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:55:10.090246 12721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:45969/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33553
--ranger_config_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:55:10.091564 12721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:55:10.092908 12721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:55:10.104733 12726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:10.104759 12729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:10.104669 12727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:55:10.106200 12721 server_base.cc:1034] running on GCE node
I20260620 02:55:10.107005 12721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:55:10.108371 12721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:55:10.109578 12721 hybrid_clock.cc:648] HybridClock initialized: now 1781924110109541 us; error 31 us; skew 500 ppm
I20260620 02:55:10.109933 12721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:55:10 dist-test-slave-8p8c krb5kdc[12277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:55:10.116288 12721 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:55:10.117808 12721 webserver.cc:460] Webserver started at http://127.0.97.62:35821/ using document root <none> and password file <none>
I20260620 02:55:10.118541 12721 fs_manager.cc:359] Metadata directory not provided
I20260620 02:55:10.118667 12721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:55:10.119004 12721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 02:55:10 dist-test-slave-8p8c krb5kdc[12277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:55:10.176831 12721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "9bf06e47566243ad8538f7900efbbe49"
format_stamp: "Formatted at 2026-06-20 02:55:10 on dist-test-slave-8p8c"
server_key: "b957e8244e7545855e787aee5f5dc314"
server_key_iv: "34dc785c7f2b5a34cf20685b9e057d5a"
server_key_version: "kuduclusterkey@0"
I20260620 02:55:10.177722 12721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "9bf06e47566243ad8538f7900efbbe49"
format_stamp: "Formatted at 2026-06-20 02:55:10 on dist-test-slave-8p8c"
server_key: "b957e8244e7545855e787aee5f5dc314"
server_key_iv: "34dc785c7f2b5a34cf20685b9e057d5a"
server_key_version: "kuduclusterkey@0"
I20260620 02:55:10.183290 12721 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 02:55:10.199455 12736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:10.201053 12721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 02:55:10.201252 12721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "9bf06e47566243ad8538f7900efbbe49"
format_stamp: "Formatted at 2026-06-20 02:55:10 on dist-test-slave-8p8c"
server_key: "b957e8244e7545855e787aee5f5dc314"
server_key_iv: "34dc785c7f2b5a34cf20685b9e057d5a"
server_key_version: "kuduclusterkey@0"
I20260620 02:55:10.201491 12721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:55:10.224798 12721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:55:10.229157 12721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:55:10.229478 12721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:55:10.248857 12721 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33553
I20260620 02:55:10.248884 12787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33553 every 8 connection(s)
I20260620 02:55:10.250257 12721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:55:10.253392 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 12721
I20260620 02:55:10.253619 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:55:10.254441 388 external_mini_cluster.cc:1428] Setting key 937dc20e645f6faf745250c47577e93e
I20260620 02:55:10.254843 12788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:55:10.270280 12788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49: Bootstrap starting.
Jun 20 02:55:10 dist-test-slave-8p8c krb5kdc[12277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:55:10.274485 12788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49: Neither blocks nor log segments found. Creating new log.
I20260620 02:55:10.275867 12788 log.cc:824] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49: Log is configured to *not* fsync() on all Append() calls
I20260620 02:55:10.279453 12788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49: No bootstrap required, opened a new log
I20260620 02:55:10.280470 12791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:10.256642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58210 (local address 127.0.97.62:33553)
0620 02:55:10.257402 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:10.257413 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:10.257459 (+ 46us) server_negotiation.cc:407] Connection header received
0620 02:55:10.259866 (+ 2407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:10.259889 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:10.260319 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:10.260691 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:10.262551 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:10.263938 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:10.265820 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:10.266825 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:10.266978 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:10.271702 (+ 4724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:10.271715 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:10.271743 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:10.271807 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:10.275771 (+ 3964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:10.276657 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:10.276664 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:10.276670 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:10.276831 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:10.277296 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:10.277301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:10.277303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:10.277962 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:10.278255 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:10.278716 (+ 461us) server_negotiation.cc:299] Negotiation successful
0620 02:55:10.279117 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":371,"threads_started":1}
I20260620 02:55:10.287585 12788 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf06e47566243ad8538f7900efbbe49" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33553 } }
I20260620 02:55:10.287904 12788 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:55:10.287966 12788 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bf06e47566243ad8538f7900efbbe49, State: Initialized, Role: FOLLOWER
I20260620 02:55:10.288442 12788 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf06e47566243ad8538f7900efbbe49" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33553 } }
I20260620 02:55:10.288635 12788 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:55:10.288786 12788 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:55:10.288944 12788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:55:10.291256 12788 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf06e47566243ad8538f7900efbbe49" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33553 } }
I20260620 02:55:10.291726 12788 leader_election.cc:302] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bf06e47566243ad8538f7900efbbe49; no voters:
I20260620 02:55:10.292306 12788 leader_election.cc:288] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:55:10.292472 12793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:55:10.293110 12793 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [term 1 LEADER]: Becoming Leader. State: Replica: 9bf06e47566243ad8538f7900efbbe49, State: Running, Role: LEADER
I20260620 02:55:10.293633 12793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf06e47566243ad8538f7900efbbe49" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33553 } }
I20260620 02:55:10.294478 12788 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:55:10.296481 12795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bf06e47566243ad8538f7900efbbe49. Latest consensus state: current_term: 1 leader_uuid: "9bf06e47566243ad8538f7900efbbe49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf06e47566243ad8538f7900efbbe49" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33553 } } }
I20260620 02:55:10.296810 12795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [sys.catalog]: This master's current role is: LEADER
I20260620 02:55:10.298493 12799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:55:10.299345 12794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bf06e47566243ad8538f7900efbbe49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf06e47566243ad8538f7900efbbe49" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33553 } } }
I20260620 02:55:10.299642 12794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [sys.catalog]: This master's current role is: LEADER
I20260620 02:55:10.303483 12799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:55:10.308548 12788 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 02:55:10.320988 12799 catalog_manager.cc:1347] Generated new cluster ID: 9fe3ed5650fa4289b2d176f800ae6775
I20260620 02:55:10.321204 12799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:55:10.344103 12799 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:55:10.345450 12799 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:55:10.357473 12799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49: Generated new TSK 0
I20260620 02:55:10.358646 12799 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:55:11.103082 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45969/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33553
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:55:11 dist-test-slave-8p8c krb5kdc[12277](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781924111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 02:55:11.512979 12833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:55:11.513312 12833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:55:11.513477 12833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:55:11.521005 12833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:55:11.521167 12833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:55:11.521358 12833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:55:11.531718 12833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:45969/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:55:11.533157 12833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:55:11.534708 12833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:55:11.552452 12844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:11.552758 12843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:11.553579 12846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:55:11.554749 12833 server_base.cc:1034] running on GCE node
I20260620 02:55:11.555493 12833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:55:11.556721 12833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:55:11.558028 12833 hybrid_clock.cc:648] HybridClock initialized: now 1781924111557922 us; error 89 us; skew 500 ppm
I20260620 02:55:11.558419 12833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:55:11 dist-test-slave-8p8c krb5kdc[12277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:55:11.562582 12833 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:55:11.564477 12833 webserver.cc:460] Webserver started at http://127.0.97.1:41919/ using document root <none> and password file <none>
I20260620 02:55:11.565426 12833 fs_manager.cc:359] Metadata directory not provided
I20260620 02:55:11.565579 12833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:55:11.565956 12833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 02:55:11 dist-test-slave-8p8c krb5kdc[12277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:55:11.592525 12833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-0/data/instance:
uuid: "88f69f8286aa473da9efc90ecb187cbe"
format_stamp: "Formatted at 2026-06-20 02:55:11 on dist-test-slave-8p8c"
server_key: "8fe2108544c2ca6b8f253a804f462d71"
server_key_iv: "7c26016d6d075dc97505ab397db432c6"
server_key_version: "kuduclusterkey@0"
I20260620 02:55:11.593535 12833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-0/wal/instance:
uuid: "88f69f8286aa473da9efc90ecb187cbe"
format_stamp: "Formatted at 2026-06-20 02:55:11 on dist-test-slave-8p8c"
server_key: "8fe2108544c2ca6b8f253a804f462d71"
server_key_iv: "7c26016d6d075dc97505ab397db432c6"
server_key_version: "kuduclusterkey@0"
I20260620 02:55:11.600001 12833 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 02:55:11.627080 12853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:11.629163 12833 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.005s
I20260620 02:55:11.629362 12833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-0/wal
uuid: "88f69f8286aa473da9efc90ecb187cbe"
format_stamp: "Formatted at 2026-06-20 02:55:11 on dist-test-slave-8p8c"
server_key: "8fe2108544c2ca6b8f253a804f462d71"
server_key_iv: "7c26016d6d075dc97505ab397db432c6"
server_key_version: "kuduclusterkey@0"
I20260620 02:55:11.629652 12833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:55:11.660483 12833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:55:11.666072 12833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:55:11.666460 12833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:55:11.667778 12833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:55:11.669682 12833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:55:11.669792 12833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:11.669967 12833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:55:11.670043 12833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:11.733624 12833 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36803
I20260620 02:55:11.733636 12965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36803 every 8 connection(s)
I20260620 02:55:11.735306 12833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:55:11.741140 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 12833
I20260620 02:55:11.741387 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:55:11.742391 388 external_mini_cluster.cc:1428] Setting key a5c83aaf6ee8e041a50f10aa656c075b
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Jun 20 02:55:11 dist-test-slave-8p8c krb5kdc[12277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:55:11.789849 12968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:11.738282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56403 (local address 127.0.97.62:33553)
0620 02:55:11.753484 (+ 15202us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:11.753494 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:11.757869 (+ 4375us) server_negotiation.cc:407] Connection header received
0620 02:55:11.757955 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:11.757962 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:11.758073 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:11.758211 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:11.759915 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:11.761210 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:11.764241 (+ 3031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:11.765091 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:11.765237 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:11.780384 (+ 15147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:11.780393 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:11.780399 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:11.780449 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:11.784579 (+ 4130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:11.785777 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:11.785783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:11.785788 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:11.785955 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:11.786611 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:11.786617 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:11.786621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:11.787026 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:11.789021 (+ 1995us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:11.789191 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 02:55:11.789434 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15026,"thread_start_us":19429,"threads_started":1}
I20260620 02:55:11.790189 12969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:11.738743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33553 (local address 127.0.97.1:56403)
0620 02:55:11.754237 (+ 15494us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:11.754315 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:11.757251 (+ 2936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:11.758543 (+ 1292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:11.758554 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:11.759021 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:55:11.759681 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:11.759704 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:11.761398 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:11.761406 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:11.764019 (+ 2613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:11.764056 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:11.765564 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:11.765574 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:11.765922 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:11.774627 (+ 8705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:55:11.774656 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:55:11.779983 (+ 5327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:55:11.784818 (+ 4835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:11.784830 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:11.784852 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:11.785557 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:11.786101 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:11.786107 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:11.786112 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:11.786445 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:11.787193 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:55:11.787204 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:55:11.787687 (+ 483us) client_negotiation.cc:769] Sending connection context
0620 02:55:11.788000 (+ 313us) client_negotiation.cc:240] Negotiation successful
0620 02:55:11.788609 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15150,"thread_start_us":14922,"threads_started":1}
I20260620 02:55:11.792629 12966 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33553
I20260620 02:55:11.792985 12966 heartbeater.cc:464] Registering TS with master...
I20260620 02:55:11.793864 12966 heartbeater.cc:511] Master 127.0.97.62:33553 requested a full tablet report, sending...
I20260620 02:55:11.796804 12753 ts_manager.cc:194] Registered new tserver with Master: 88f69f8286aa473da9efc90ecb187cbe (127.0.97.1:36803)
I20260620 02:55:11.799100 12753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:55:11.818815 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45969/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33553
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:55:11 dist-test-slave-8p8c krb5kdc[12277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260620 02:55:12.285267 12972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:55:12.285804 12972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:55:12.286027 12972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:55:12.298739 12972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:55:12.299022 12972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:55:12.299405 12972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:55:12.315423 12972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:45969/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:55:12.317425 12972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:55:12.319181 12972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Jun 20 02:55:12 dist-test-slave-8p8c krb5kdc[12277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260620 02:55:12.346256 12979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:12.347277 12978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:55:12.354676 12972 server_base.cc:1034] running on GCE node
W20260620 02:55:12.356137 12982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:55:12.360381 12972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:55:12.378917 12972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:55:12.380431 12972 hybrid_clock.cc:648] HybridClock initialized: now 1781924112380375 us; error 42 us; skew 500 ppm
I20260620 02:55:12.380894 12972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:55:12 dist-test-slave-8p8c krb5kdc[12277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:55:12.386435 12972 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:55:12.423316 12972 webserver.cc:460] Webserver started at http://127.0.97.2:39369/ using document root <none> and password file <none>
I20260620 02:55:12.424528 12972 fs_manager.cc:359] Metadata directory not provided
I20260620 02:55:12.424757 12972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:55:12.425197 12972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 02:55:12 dist-test-slave-8p8c krb5kdc[12277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:55:12.451906 12972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-1/data/instance:
uuid: "ce9df07e95fd4579bbc110f584cd4abf"
format_stamp: "Formatted at 2026-06-20 02:55:12 on dist-test-slave-8p8c"
server_key: "d7f1ffafc07309d2b7b5504c8d57bcf5"
server_key_iv: "8d28ff2d1f848dd96e86a7653687ea9e"
server_key_version: "kuduclusterkey@0"
I20260620 02:55:12.453075 12972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ce9df07e95fd4579bbc110f584cd4abf"
format_stamp: "Formatted at 2026-06-20 02:55:12 on dist-test-slave-8p8c"
server_key: "d7f1ffafc07309d2b7b5504c8d57bcf5"
server_key_iv: "8d28ff2d1f848dd96e86a7653687ea9e"
server_key_version: "kuduclusterkey@0"
I20260620 02:55:12.461552 12972 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.004s
I20260620 02:55:12.504622 12989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:12.506954 12972 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.005s sys 0.002s
I20260620 02:55:12.507256 12972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-1/wal
uuid: "ce9df07e95fd4579bbc110f584cd4abf"
format_stamp: "Formatted at 2026-06-20 02:55:12 on dist-test-slave-8p8c"
server_key: "d7f1ffafc07309d2b7b5504c8d57bcf5"
server_key_iv: "8d28ff2d1f848dd96e86a7653687ea9e"
server_key_version: "kuduclusterkey@0"
I20260620 02:55:12.507624 12972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:55:12.573422 12972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:55:12.578771 12972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:55:12.579198 12972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:55:12.586628 12972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:55:12.593647 12972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:55:12.593781 12972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:12.593919 12972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:55:12.593995 12972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:12.662210 12972 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41733
I20260620 02:55:12.662231 13103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41733 every 8 connection(s)
I20260620 02:55:12.663771 12972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:55:12.665361 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 12972
I20260620 02:55:12.665548 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:55:12.666116 388 external_mini_cluster.cc:1428] Setting key fddbd585ea5923f89d9f7a66a77d96df
Jun 20 02:55:12 dist-test-slave-8p8c krb5kdc[12277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:55:12.694319 13109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:12.668227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52977 (local address 127.0.97.62:33553)
0620 02:55:12.674158 (+ 5931us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:12.674165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:12.674186 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:55:12.674246 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:12.674251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:12.674328 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:12.674478 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:12.676775 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:12.677971 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:12.680684 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:12.681311 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:12.681441 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:12.686159 (+ 4718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:12.686166 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:12.686171 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:12.686214 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:12.689844 (+ 3630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:12.690954 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:12.690963 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:12.690969 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:12.691137 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:12.691841 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:12.691849 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:12.691855 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:12.692275 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:12.692436 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:12.693423 (+ 987us) server_negotiation.cc:299] Negotiation successful
0620 02:55:12.693809 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5806,"thread_start_us":5691,"threads_started":1}
I20260620 02:55:12.695297 13110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:12.668296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33553 (local address 127.0.97.2:52977)
0620 02:55:12.669450 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:12.669515 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:12.672828 (+ 3313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:12.674924 (+ 2096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:12.674938 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:12.675609 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:55:12.676534 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:12.676570 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:12.678165 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:12.678173 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:12.680500 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:12.680518 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:12.681573 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:12.681583 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:12.681926 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:12.682774 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:55:12.682806 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:55:12.685894 (+ 3088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:55:12.690028 (+ 4134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:12.690041 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:12.690064 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:12.690782 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:12.691280 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:12.691288 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:12.691294 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:12.691682 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:12.692430 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:55:12.692440 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:55:12.692946 (+ 506us) client_negotiation.cc:769] Sending connection context
0620 02:55:12.693269 (+ 323us) client_negotiation.cc:240] Negotiation successful
0620 02:55:12.693839 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"thread_start_us":619,"threads_started":1}
I20260620 02:55:12.696954 13104 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33553
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:55:12.697306 13104 heartbeater.cc:464] Registering TS with master...
I20260620 02:55:12.698050 13104 heartbeater.cc:511] Master 127.0.97.62:33553 requested a full tablet report, sending...
I20260620 02:55:12.699796 12753 ts_manager.cc:194] Registered new tserver with Master: ce9df07e95fd4579bbc110f584cd4abf (127.0.97.2:41733)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 02:55:12.700923 12753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:55:12.731185 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45969/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33553
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:55:12.803469 12966 heartbeater.cc:503] Master 127.0.97.62:33553 was elected leader, sending a full tablet report...
W20260620 02:55:13.084578 13115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:55:13.084853 13115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:55:13.085007 13115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:55:13.092185 13115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:55:13.092347 13115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:55:13.092514 13115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:55:13.103412 13115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:45969/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:55:13.104830 13115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:55:13.106351 13115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:55:13.119622 13120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:13.119719 13121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:13.119670 13123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:55:13.123975 13115 server_base.cc:1034] running on GCE node
I20260620 02:55:13.124665 13115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:55:13.125666 13115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:55:13.126945 13115 hybrid_clock.cc:648] HybridClock initialized: now 1781924113126897 us; error 39 us; skew 500 ppm
I20260620 02:55:13.127339 13115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:55:13 dist-test-slave-8p8c krb5kdc[12277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:55:13.130723 13115 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:55:13.132205 13115 webserver.cc:460] Webserver started at http://127.0.97.3:40945/ using document root <none> and password file <none>
I20260620 02:55:13.132920 13115 fs_manager.cc:359] Metadata directory not provided
I20260620 02:55:13.133064 13115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:55:13.133364 13115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 02:55:13 dist-test-slave-8p8c krb5kdc[12277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:55:13.151362 13115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-2/data/instance:
uuid: "3ed297b3acbc41619e78947a13848a00"
format_stamp: "Formatted at 2026-06-20 02:55:13 on dist-test-slave-8p8c"
server_key: "a5e7318067224dbb35fae22146551a7e"
server_key_iv: "fcdaed3e5c480ea6d0f23d5a54c16ba5"
server_key_version: "kuduclusterkey@0"
I20260620 02:55:13.152473 13115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3ed297b3acbc41619e78947a13848a00"
format_stamp: "Formatted at 2026-06-20 02:55:13 on dist-test-slave-8p8c"
server_key: "a5e7318067224dbb35fae22146551a7e"
server_key_iv: "fcdaed3e5c480ea6d0f23d5a54c16ba5"
server_key_version: "kuduclusterkey@0"
I20260620 02:55:13.159678 13115 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260620 02:55:13.176823 13130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:13.178622 13115 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:55:13.178838 13115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-2/wal
uuid: "3ed297b3acbc41619e78947a13848a00"
format_stamp: "Formatted at 2026-06-20 02:55:13 on dist-test-slave-8p8c"
server_key: "a5e7318067224dbb35fae22146551a7e"
server_key_iv: "fcdaed3e5c480ea6d0f23d5a54c16ba5"
server_key_version: "kuduclusterkey@0"
I20260620 02:55:13.179075 13115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:55:13.199878 13115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:55:13.204198 13115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:55:13.204517 13115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:55:13.205607 13115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:55:13.207158 13115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:55:13.207267 13115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:13.207412 13115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:55:13.207480 13115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:13.243162 13115 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44579
I20260620 02:55:13.243193 13242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44579 every 8 connection(s)
I20260620 02:55:13.244727 13115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:55:13.248771 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 13115
I20260620 02:55:13.248945 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:55:13.249523 388 external_mini_cluster.cc:1428] Setting key 8fcd1baa4d0867911fd0c80b6c7f3054
Jun 20 02:55:13 dist-test-slave-8p8c krb5kdc[12277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:55:13.268033 13245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:13.247576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53491 (local address 127.0.97.62:33553)
0620 02:55:13.248142 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:13.248147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:13.249310 (+ 1163us) server_negotiation.cc:407] Connection header received
0620 02:55:13.251902 (+ 2592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:13.251907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:13.252021 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:13.252138 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:13.253957 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:13.255055 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:13.257389 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:13.257851 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:13.257977 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:13.261558 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:13.261563 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:13.261567 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:13.261599 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:13.265021 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:13.265768 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:13.265772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:13.265774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:13.265892 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:13.266298 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:13.266301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:13.266303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:13.266707 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:13.266836 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:13.267591 (+ 755us) server_negotiation.cc:299] Negotiation successful
0620 02:55:13.267752 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":269,"threads_started":1}
I20260620 02:55:13.268957 13246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:13.248118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33553 (local address 127.0.97.3:53491)
0620 02:55:13.249138 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:13.249187 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:13.251606 (+ 2419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:13.252398 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:13.252410 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:13.252970 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:55:13.253778 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:13.253802 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:13.255212 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:13.255217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:13.257235 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:13.257250 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:13.258069 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:13.258075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:13.258380 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:13.259139 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:55:13.259167 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:55:13.261309 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:55:13.265200 (+ 3891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:13.265210 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:13.265229 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:13.265656 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:13.266011 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:13.266014 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:13.266017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:13.266179 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:13.266838 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:55:13.266845 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:55:13.267190 (+ 345us) client_negotiation.cc:769] Sending connection context
0620 02:55:13.267501 (+ 311us) client_negotiation.cc:240] Negotiation successful
0620 02:55:13.267877 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":466,"threads_started":1}
I20260620 02:55:13.270315 13243 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33553
I20260620 02:55:13.270653 13243 heartbeater.cc:464] Registering TS with master...
I20260620 02:55:13.271339 13243 heartbeater.cc:511] Master 127.0.97.62:33553 requested a full tablet report, sending...
I20260620 02:55:13.272747 12753 ts_manager.cc:194] Registered new tserver with Master: 3ed297b3acbc41619e78947a13848a00 (127.0.97.3:44579)
I20260620 02:55:13.273571 12753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53491
I20260620 02:55:13.277397 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:55:13.279093 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 12833
Jun 20 02:55:13 dist-test-slave-8p8c krb5kdc[12277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:55:13.295233 13248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:13.280062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39336 (local address 127.0.97.1:36803)
0620 02:55:13.280716 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:13.280722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:13.280745 (+ 23us) server_negotiation.cc:407] Connection header received
0620 02:55:13.280796 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:13.280801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:13.280980 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:13.281098 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:13.282533 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:13.283583 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:13.285002 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:13.285530 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:13.285621 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:13.289693 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:13.289703 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:13.289716 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:13.289747 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:13.292416 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:13.293093 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:13.293100 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:13.293102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:13.293268 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:13.293607 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:13.293610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:13.293612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:13.293906 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:13.294121 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:13.294753 (+ 632us) server_negotiation.cc:299] Negotiation successful
0620 02:55:13.294937 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":331,"threads_started":1}
I20260620 02:55:13.296504 12961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:55:13.380450 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 12833
I20260620 02:55:13.396632 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 12972
Jun 20 02:55:13 dist-test-slave-8p8c krb5kdc[12277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:55:13.414022 13250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:13.397558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39046 (local address 127.0.97.2:41733)
0620 02:55:13.398099 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:13.398106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:13.398132 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:55:13.398200 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:13.398206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:13.398475 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:13.398653 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:13.399946 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:13.400947 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:13.402327 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:13.403100 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:13.403241 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:13.407998 (+ 4757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:13.408009 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:13.408025 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:13.408065 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:13.411063 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:13.411841 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:13.411849 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:13.411851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:13.411995 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:13.412372 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:13.412377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:13.412380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:13.412729 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:13.413070 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:13.413363 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 02:55:13.413640 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":260,"threads_started":1}
I20260620 02:55:13.415424 13097 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:55:13.498790 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 12972
I20260620 02:55:13.514851 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 13115
Jun 20 02:55:13 dist-test-slave-8p8c krb5kdc[12277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:55:13.532188 13252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:13.515860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52160 (local address 127.0.97.3:44579)
0620 02:55:13.516464 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:13.516471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:13.516501 (+ 30us) server_negotiation.cc:407] Connection header received
0620 02:55:13.516584 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:13.516592 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:13.516814 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:13.516940 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:13.518336 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:13.519440 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:13.520765 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:13.521316 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:13.521412 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:13.525906 (+ 4494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:13.525917 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:13.525933 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:13.525969 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:13.528942 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:13.529664 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:13.529672 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:13.529674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:13.529817 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:13.530265 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:13.530269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:13.530272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:13.530902 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:13.531218 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:13.531548 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 02:55:13.531752 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":297,"threads_started":1}
I20260620 02:55:13.533404 13238 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:55:13.617345 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 13115
I20260620 02:55:13.633549 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 12721
I20260620 02:55:13.634980 12783 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:55:13.743994 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 12721
I20260620 02:55:13.765390 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33553
--webserver_interface=127.0.97.62
--webserver_port=35821
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33553
--ranger_config_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45969/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:45969/kms with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:55:14.101102 13255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:55:14.101384 13255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:55:14.101469 13255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:55:14.108493 13255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:55:14.108597 13255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:55:14.108642 13255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:55:14.108698 13255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:55:14.108745 13255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:55:14.118614 13255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:45969/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33553
--ranger_config_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=35821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:55:14.119836 13255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:55:14.121083 13255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:55:14.132927 13263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:14.132966 13261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:14.133006 13260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:55:14.133708 13255 server_base.cc:1034] running on GCE node
I20260620 02:55:14.134536 13255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:55:14.135895 13255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:55:14.137117 13255 hybrid_clock.cc:648] HybridClock initialized: now 1781924114137033 us; error 65 us; skew 500 ppm
I20260620 02:55:14.137472 13255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:55:14 dist-test-slave-8p8c krb5kdc[12277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:55:14.140647 13255 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:55:14.142182 13255 webserver.cc:460] Webserver started at http://127.0.97.62:35821/ using document root <none> and password file <none>
I20260620 02:55:14.142941 13255 fs_manager.cc:359] Metadata directory not provided
I20260620 02:55:14.143087 13255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 02:55:14 dist-test-slave-8p8c krb5kdc[12277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:55:14.193077 13255 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:55:14.196640 13271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:14.198120 13255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:55:14.198323 13255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "9bf06e47566243ad8538f7900efbbe49"
format_stamp: "Formatted at 2026-06-20 02:55:10 on dist-test-slave-8p8c"
server_key: "b957e8244e7545855e787aee5f5dc314"
server_key_iv: "34dc785c7f2b5a34cf20685b9e057d5a"
server_key_version: "kuduclusterkey@0"
I20260620 02:55:14.199388 13255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:55:14.231248 13255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:55:14.235592 13255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:55:14.235895 13255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:55:14.255208 13255 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33553
I20260620 02:55:14.255247 13322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33553 every 8 connection(s)
I20260620 02:55:14.256734 13255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:55:14.260524 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 13255
I20260620 02:55:14.261616 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45969/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:36803
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:33553
--webserver_port=41919
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:45969/kms with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:55:14.266438 13323 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:55:14.271734 13323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49: Bootstrap starting.
I20260620 02:55:14.291545 13323 log.cc:824] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49: Log is configured to *not* fsync() on all Append() calls
I20260620 02:55:14.300357 13323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:55:14.300920 13323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49: Bootstrap complete.
I20260620 02:55:14.310303 13323 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf06e47566243ad8538f7900efbbe49" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33553 } }
I20260620 02:55:14.311234 13323 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bf06e47566243ad8538f7900efbbe49, State: Initialized, Role: FOLLOWER
I20260620 02:55:14.311961 13323 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf06e47566243ad8538f7900efbbe49" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33553 } }
I20260620 02:55:14.312196 13323 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:55:14.312307 13323 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:55:14.312500 13323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:55:14.316327 13323 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf06e47566243ad8538f7900efbbe49" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33553 } }
I20260620 02:55:14.316809 13323 leader_election.cc:302] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bf06e47566243ad8538f7900efbbe49; no voters:
I20260620 02:55:14.317478 13323 leader_election.cc:288] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 02:55:14.317627 13327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 02:55:14.318801 13327 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [term 2 LEADER]: Becoming Leader. State: Replica: 9bf06e47566243ad8538f7900efbbe49, State: Running, Role: LEADER
I20260620 02:55:14.319412 13327 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf06e47566243ad8538f7900efbbe49" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33553 } }
I20260620 02:55:14.319729 13323 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:55:14.321961 13329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bf06e47566243ad8538f7900efbbe49. Latest consensus state: current_term: 2 leader_uuid: "9bf06e47566243ad8538f7900efbbe49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf06e47566243ad8538f7900efbbe49" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33553 } } }
I20260620 02:55:14.332842 13329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [sys.catalog]: This master's current role is: LEADER
I20260620 02:55:14.327358 13328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9bf06e47566243ad8538f7900efbbe49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf06e47566243ad8538f7900efbbe49" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33553 } } }
I20260620 02:55:14.334719 13328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49 [sys.catalog]: This master's current role is: LEADER
I20260620 02:55:14.346536 13333 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:55:14.353533 13333 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:55:14.355042 13333 catalog_manager.cc:1259] Loaded cluster ID: 9fe3ed5650fa4289b2d176f800ae6775
I20260620 02:55:14.355208 13333 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:55:14.359026 13333 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:55:14.360878 13333 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49: Loaded TSK: 0
I20260620 02:55:14.362550 13333 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:55:14.370997 13323 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260620 02:55:14.687175 13325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:55:14.687727 13325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:55:14.687975 13325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:55:14.699613 13325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:55:14.699853 13325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:55:14.700088 13325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:55:14.716569 13325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:45969/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:36803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=41919
--tserver_master_addrs=127.0.97.62:33553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:55:14.718853 13325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:55:14.720714 13325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:55:14.751680 13362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:14.761656 13364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:14.762303 13361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:55:14.762598 13325 server_base.cc:1034] running on GCE node
I20260620 02:55:14.763767 13325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:55:14.765581 13325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:55:14.766899 13325 hybrid_clock.cc:648] HybridClock initialized: now 1781924114766784 us; error 107 us; skew 500 ppm
I20260620 02:55:14.767392 13325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:55:14 dist-test-slave-8p8c krb5kdc[12277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:55:14.772114 13325 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:55:14.788040 13325 webserver.cc:460] Webserver started at http://127.0.97.1:41919/ using document root <none> and password file <none>
I20260620 02:55:14.789203 13325 fs_manager.cc:359] Metadata directory not provided
I20260620 02:55:14.789484 13325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 02:55:14 dist-test-slave-8p8c krb5kdc[12277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:55:14.824898 13325 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:55:14.828792 13372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:14.831063 13325 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 02:55:14.831310 13325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-0/wal
uuid: "88f69f8286aa473da9efc90ecb187cbe"
format_stamp: "Formatted at 2026-06-20 02:55:11 on dist-test-slave-8p8c"
server_key: "8fe2108544c2ca6b8f253a804f462d71"
server_key_iv: "7c26016d6d075dc97505ab397db432c6"
server_key_version: "kuduclusterkey@0"
I20260620 02:55:14.832782 13325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:55:14.880050 13325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:55:14.885641 13325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:55:14.886014 13325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:55:14.897794 13325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:55:14.899885 13325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:55:14.900079 13325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 02:55:14.900206 13325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:55:14.900321 13325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:15.000437 13325 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36803
I20260620 02:55:15.000499 13485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36803 every 8 connection(s)
I20260620 02:55:15.002573 13325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:55:15.008683 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 13325
I20260620 02:55:15.009438 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45969/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:41733
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:33553
--webserver_port=39369
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:45969/kms with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:55:15 dist-test-slave-8p8c krb5kdc[12277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:55:15.046118 13489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:15.007275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33553 (local address 127.0.97.1:35611)
0620 02:55:15.008228 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:15.008284 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:15.010882 (+ 2598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:15.017036 (+ 6154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:15.017051 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:15.017721 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:55:15.018613 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:15.018644 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:15.021168 (+ 2524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:15.021176 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:15.023530 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:15.023548 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:15.025217 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:15.025226 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:15.025561 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:15.026468 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:55:15.026507 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:55:15.029506 (+ 2999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:55:15.040189 (+ 10683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:15.040202 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:15.040220 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:15.040916 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:15.041577 (+ 661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:15.041582 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:15.041586 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:15.041838 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:15.043010 (+ 1172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:55:15.043019 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:55:15.043532 (+ 513us) client_negotiation.cc:769] Sending connection context
0620 02:55:15.043887 (+ 355us) client_negotiation.cc:240] Negotiation successful
0620 02:55:15.044642 (+ 755us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":385,"threads_started":1}
I20260620 02:55:15.048970 13488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:15.007086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35611 (local address 127.0.97.62:33553)
0620 02:55:15.015057 (+ 7971us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:15.015074 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:15.015125 (+ 51us) server_negotiation.cc:407] Connection header received
0620 02:55:15.015723 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:15.015749 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:15.016323 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:15.016796 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:15.018854 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:15.020976 (+ 2122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:15.023722 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:15.024897 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:15.025136 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:15.029846 (+ 4710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:15.029858 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:15.029879 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:15.029937 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:15.039933 (+ 9996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:15.041134 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:15.041146 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:15.041157 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:15.041390 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:15.042026 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:15.042033 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:15.042038 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:15.042835 (+ 797us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:15.046837 (+ 4002us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:15.047139 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 02:55:15.047600 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7649,"thread_start_us":7355,"threads_started":1}
I20260620 02:55:15.052376 13486 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33553
I20260620 02:55:15.052709 13486 heartbeater.cc:464] Registering TS with master...
I20260620 02:55:15.053598 13486 heartbeater.cc:511] Master 127.0.97.62:33553 requested a full tablet report, sending...
W20260620 02:55:15.056954 13486 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:33553 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 02:55:15.057278 13486 heartbeater.cc:464] Registering TS with master...
I20260620 02:55:15.057636 13486 heartbeater.cc:511] Master 127.0.97.62:33553 requested a full tablet report, sending...
I20260620 02:55:15.059072 13486 heartbeater.cc:464] Registering TS with master...
I20260620 02:55:15.059412 13486 heartbeater.cc:511] Master 127.0.97.62:33553 requested a full tablet report, sending...
W20260620 02:55:15.060591 13486 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 02:55:15.269224 13497 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260620 02:55:15.432854 13490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:55:15.433291 13490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:55:15.433506 13490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:55:15.442276 13490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:55:15.442451 13490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:55:15.442651 13490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:55:15.453222 13490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:45969/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:41733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=39369
--tserver_master_addrs=127.0.97.62:33553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:55:15.454708 13490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:55:15.456048 13490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:55:15.474872 13504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:15.476013 13505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:15.479174 13507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:55:15.479604 13490 server_base.cc:1034] running on GCE node
I20260620 02:55:15.480295 13490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:55:15.481329 13490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:55:15.482585 13490 hybrid_clock.cc:648] HybridClock initialized: now 1781924115482528 us; error 39 us; skew 500 ppm
I20260620 02:55:15.482970 13490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:55:15 dist-test-slave-8p8c krb5kdc[12277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:55:15.490875 13490 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:55:15.520691 13490 webserver.cc:460] Webserver started at http://127.0.97.2:39369/ using document root <none> and password file <none>
I20260620 02:55:15.521835 13490 fs_manager.cc:359] Metadata directory not provided
I20260620 02:55:15.522024 13490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 02:55:15 dist-test-slave-8p8c krb5kdc[12277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:55:15.559983 13490 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 02:55:15.563894 13517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:15.565836 13490 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:55:15.566030 13490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-1/wal
uuid: "ce9df07e95fd4579bbc110f584cd4abf"
format_stamp: "Formatted at 2026-06-20 02:55:12 on dist-test-slave-8p8c"
server_key: "d7f1ffafc07309d2b7b5504c8d57bcf5"
server_key_iv: "8d28ff2d1f848dd96e86a7653687ea9e"
server_key_version: "kuduclusterkey@0"
I20260620 02:55:15.567482 13490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:55:15.592304 13490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:55:15.597811 13490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:55:15.598145 13490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:55:15.599470 13490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:55:15.601303 13490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:55:15.601411 13490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:15.601518 13490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:55:15.601598 13490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 02:55:15 dist-test-slave-8p8c krb5kdc[12277](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781924115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:55:15.689358 13490 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41733
I20260620 02:55:15.689397 13629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41733 every 8 connection(s)
I20260620 02:55:15.691241 13490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:55:15.693674 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 13490
I20260620 02:55:15.694568 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45969/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:44579
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:33553
--webserver_port=40945
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:45969/kms with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:55:15 dist-test-slave-8p8c krb5kdc[12277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:55:15.737188 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:15.694210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51795 (local address 127.0.97.62:33553)
0620 02:55:15.710121 (+ 15911us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:15.710129 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:15.715246 (+ 5117us) server_negotiation.cc:407] Connection header received
0620 02:55:15.717476 (+ 2230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:15.717483 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:15.717587 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:15.717710 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:15.720372 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:15.721448 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:15.723643 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:15.724290 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:15.724396 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:15.729526 (+ 5130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:15.729534 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:15.729540 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:15.729587 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:15.733286 (+ 3699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:15.734291 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:15.734296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:15.734301 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:15.734491 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:15.735029 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:15.735035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:15.735039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:15.735413 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:15.735555 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:15.736567 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0620 02:55:15.736805 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15749,"thread_start_us":20874,"threads_started":1}
I20260620 02:55:15.739981 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:15.694882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33553 (local address 127.0.97.2:51795)
0620 02:55:15.710799 (+ 15917us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:15.710861 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:15.717041 (+ 6180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:15.718266 (+ 1225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:15.718281 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:15.719045 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:55:15.720129 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:15.720161 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:15.721635 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:15.721639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:15.723457 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:15.723473 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:15.724758 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:15.724766 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:15.725088 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:15.726200 (+ 1112us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:55:15.726240 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:55:15.729219 (+ 2979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:55:15.733469 (+ 4250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:15.733481 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:15.733501 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:15.734131 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:15.734651 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:15.734656 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:15.734660 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:15.734898 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:15.735647 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:55:15.735656 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:55:15.736093 (+ 437us) client_negotiation.cc:769] Sending connection context
0620 02:55:15.737792 (+ 1699us) client_negotiation.cc:240] Negotiation successful
0620 02:55:15.738468 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15555,"thread_start_us":15314,"threads_started":1}
I20260620 02:55:15.741998 13630 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33553
I20260620 02:55:15.742352 13630 heartbeater.cc:464] Registering TS with master...
I20260620 02:55:15.743393 13630 heartbeater.cc:511] Master 127.0.97.62:33553 requested a full tablet report, sending...
I20260620 02:55:15.745560 13288 ts_manager.cc:194] Registered new tserver with Master: ce9df07e95fd4579bbc110f584cd4abf (127.0.97.2:41733)
I20260620 02:55:15.747925 13288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51795
Jun 20 02:55:16 dist-test-slave-8p8c krb5kdc[12277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:55:16.060774 13486 heartbeater.cc:464] Registering TS with master...
I20260620 02:55:16.061261 13486 heartbeater.cc:511] Master 127.0.97.62:33553 requested a full tablet report, sending...
I20260620 02:55:16.068584 13288 ts_manager.cc:194] Registered new tserver with Master: 88f69f8286aa473da9efc90ecb187cbe (127.0.97.1:36803)
I20260620 02:55:16.069813 13288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35611
W20260620 02:55:16.213086 13634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:55:16.213521 13634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:55:16.213683 13634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:55:16.225039 13634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:55:16.225183 13634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:55:16.225349 13634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:55:16.240748 13634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:45969/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:44579
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=40945
--tserver_master_addrs=127.0.97.62:33553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:55:16.242434 13634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:55:16.244119 13634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:55:16.280310 13640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:16.282757 13641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:16.283370 13644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:55:16.283892 13634 server_base.cc:1034] running on GCE node
I20260620 02:55:16.284780 13634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:55:16.291109 13634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:55:16.292333 13634 hybrid_clock.cc:648] HybridClock initialized: now 1781924116292259 us; error 73 us; skew 500 ppm
I20260620 02:55:16.292728 13634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:55:16 dist-test-slave-8p8c krb5kdc[12277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:55:16.296885 13634 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:55:16.317708 13634 webserver.cc:460] Webserver started at http://127.0.97.3:40945/ using document root <none> and password file <none>
I20260620 02:55:16.318953 13634 fs_manager.cc:359] Metadata directory not provided
I20260620 02:55:16.319198 13634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 02:55:16 dist-test-slave-8p8c krb5kdc[12277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:55:16.372375 13634 fs_manager.cc:709] Time spent opening directory manager: real 0.017s user 0.003s sys 0.002s
I20260620 02:55:16.387193 13651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:16.389271 13634 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.006s sys 0.005s
I20260620 02:55:16.389531 13634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-2/wal
uuid: "3ed297b3acbc41619e78947a13848a00"
format_stamp: "Formatted at 2026-06-20 02:55:13 on dist-test-slave-8p8c"
server_key: "a5e7318067224dbb35fae22146551a7e"
server_key_iv: "fcdaed3e5c480ea6d0f23d5a54c16ba5"
server_key_version: "kuduclusterkey@0"
I20260620 02:55:16.390998 13634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Jun 20 02:55:16 dist-test-slave-8p8c krb5kdc[12277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 02:55:16.461637 13634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:55:16.467010 13634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:55:16.467406 13634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:55:16.468735 13634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:55:16.470628 13634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:55:16.470788 13634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:16.470942 13634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:55:16.471076 13634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:16.644016 13634 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44579
I20260620 02:55:16.646111 13634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:55:16.644060 13764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44579 every 8 connection(s)
I20260620 02:55:16.649300 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 13634
Jun 20 02:55:16 dist-test-slave-8p8c krb5kdc[12277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:55:16.693234 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:16.651291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34083 (local address 127.0.97.62:33553)
0620 02:55:16.663467 (+ 12176us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:16.663477 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:16.669045 (+ 5568us) server_negotiation.cc:407] Connection header received
0620 02:55:16.669149 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:16.669158 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:16.669293 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:16.669423 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:16.672655 (+ 3232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:16.674250 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:16.677809 (+ 3559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:16.678761 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:16.678901 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:16.685598 (+ 6697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:16.685605 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:16.685610 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:16.685650 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:16.689347 (+ 3697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:16.690457 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:16.690463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:16.690468 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:16.690599 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:16.691109 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:16.691114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:16.691118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:16.691487 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:16.691618 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:16.692563 (+ 945us) server_negotiation.cc:299] Negotiation successful
0620 02:55:16.692825 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11914,"thread_start_us":12328,"threads_started":1}
I20260620 02:55:16.695745 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:16.650562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33553 (local address 127.0.97.3:34083)
0620 02:55:16.664717 (+ 14155us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:16.664794 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:16.668483 (+ 3689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:16.670113 (+ 1630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:16.670132 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:16.671080 (+ 948us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:55:16.672370 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:16.672406 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:16.674653 (+ 2247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:16.674662 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:16.677573 (+ 2911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:16.677595 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:16.679437 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:16.679446 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:16.679886 (+ 440us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:16.681834 (+ 1948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:55:16.681868 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:55:16.685323 (+ 3455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:55:16.689623 (+ 4300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:16.689631 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:16.689650 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:16.690277 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:16.690731 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:16.690735 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:16.690739 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:16.690963 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:16.691718 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:55:16.691726 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:55:16.692149 (+ 423us) client_negotiation.cc:769] Sending connection context
0620 02:55:16.693783 (+ 1634us) client_negotiation.cc:240] Negotiation successful
0620 02:55:16.694384 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13709,"thread_start_us":13419,"threads_started":1}
I20260620 02:55:16.699195 13765 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33553
I20260620 02:55:16.699532 13765 heartbeater.cc:464] Registering TS with master...
I20260620 02:55:16.700314 13765 heartbeater.cc:511] Master 127.0.97.62:33553 requested a full tablet report, sending...
I20260620 02:55:16.701856 13288 ts_manager.cc:194] Registered new tserver with Master: 3ed297b3acbc41619e78947a13848a00 (127.0.97.3:44579)
I20260620 02:55:16.703071 13288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34083
I20260620 02:55:16.711319 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:16.651663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48860 (local address 127.0.97.62:33553)
0620 02:55:16.663915 (+ 12252us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:16.663923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:16.665793 (+ 1870us) server_negotiation.cc:407] Connection header received
0620 02:55:16.666033 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:16.666038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:16.666127 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:16.666246 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:16.680524 (+ 14278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:16.681529 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:16.683639 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:16.684138 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:16.684275 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:16.696234 (+ 11959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:16.696239 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:16.696243 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:16.696303 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:16.699749 (+ 3446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:16.706429 (+ 6680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:16.706436 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:16.706440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:16.706620 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:16.709265 (+ 2645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:16.709272 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:16.709276 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:16.709679 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:16.709975 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:16.710638 (+ 663us) server_negotiation.cc:299] Negotiation successful
0620 02:55:16.710905 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12113,"thread_start_us":11395,"threads_started":1}
I20260620 02:55:16.715586 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:55:16.738672 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:16.719703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48868 (local address 127.0.97.62:33553)
0620 02:55:16.719965 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:16.719974 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:16.722546 (+ 2572us) server_negotiation.cc:407] Connection header received
0620 02:55:16.722713 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:16.722719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:16.722815 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:16.722924 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:16.724840 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:16.725925 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:16.727926 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:16.728596 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:16.728741 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:16.731745 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:16.731751 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:16.731756 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:16.731798 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:16.735400 (+ 3602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:16.736404 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:16.736410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:16.736414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:16.736598 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:16.737007 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:16.737012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:16.737015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:16.737380 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:16.737645 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:16.737980 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 02:55:16.738280 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 02:55:16.744570 13288 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48868:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 02:55:16.753336 13630 heartbeater.cc:503] Master 127.0.97.62:33553 was elected leader, sending a full tablet report...
W20260620 02:55:16.753470 13288 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:55:16.790637 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:16.769624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41733 (local address 127.0.0.1:39060)
0620 02:55:16.781242 (+ 11618us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:16.781263 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:16.781468 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:16.782773 (+ 1305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:16.782784 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:16.782855 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:55:16.783501 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:16.783515 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:16.785433 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:16.785438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:16.787749 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:16.787777 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:16.789543 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:16.789549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:16.789757 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:16.789809 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 02:55:16.789950 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 02:55:16.790159 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11482,"mutex_wait_us":2,"spinlock_wait_cycles":4480,"thread_start_us":10264,"threads_started":1}
I20260620 02:55:16.790594 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:16.769933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39060 (local address 127.0.97.2:41733)
0620 02:55:16.779391 (+ 9458us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:16.779406 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:16.781360 (+ 1954us) server_negotiation.cc:407] Connection header received
0620 02:55:16.781713 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:16.781726 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:16.782086 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:16.782293 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:55:16.783734 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:16.785327 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:16.787964 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:16.789331 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:16.789478 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:16.789727 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:16.790042 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 02:55:16.790218 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9198,"thread_start_us":9504,"threads_started":1}
I20260620 02:55:16.791922 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:16.781298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52176 (local address 127.0.97.3:44579)
0620 02:55:16.783357 (+ 2059us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:16.783368 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:16.783999 (+ 631us) server_negotiation.cc:407] Connection header received
0620 02:55:16.784117 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:16.784127 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:16.784378 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:16.784560 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:55:16.785398 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:16.786851 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:16.789037 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:16.790355 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:16.790511 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:16.790754 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:16.791410 (+ 656us) server_negotiation.cc:299] Negotiation successful
0620 02:55:16.791586 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1882,"thread_start_us":1739,"threads_started":1}
I20260620 02:55:16.791921 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:16.781733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44579 (local address 127.0.0.1:52176)
0620 02:55:16.783705 (+ 1972us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:16.783722 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:16.783864 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:16.784834 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:16.784839 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:16.784891 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:55:16.785196 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:16.785205 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:16.787025 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:16.787046 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:16.788892 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:16.788902 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:16.790988 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:16.790994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:16.791168 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:16.791237 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 02:55:16.791338 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 02:55:16.791484 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1832,"mutex_wait_us":4,"spinlock_wait_cycles":768,"thread_start_us":1706,"threads_started":1}
I20260620 02:55:16.792455 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:16.769416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36803 (local address 127.0.0.1:39338)
0620 02:55:16.780300 (+ 10884us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:16.780343 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:16.780556 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:16.782781 (+ 2225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:16.782789 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:16.782841 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:55:16.784058 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:16.784070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:16.786014 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:16.786023 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:16.788350 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:16.788364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:16.791064 (+ 2700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:16.791070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:16.791800 (+ 730us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:16.791833 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 02:55:16.791946 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 02:55:16.792098 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10607,"mutex_wait_us":556,"spinlock_wait_cycles":6016,"thread_start_us":11548,"threads_started":1}
I20260620 02:55:16.792872 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:16.769569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39338 (local address 127.0.97.1:36803)
0620 02:55:16.781692 (+ 12123us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:16.781700 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:16.781727 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:55:16.781833 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:16.781844 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:16.782181 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:16.782452 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:55:16.784269 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:16.785855 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:16.788517 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:16.789955 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:16.790085 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:16.790311 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:16.792252 (+ 1941us) server_negotiation.cc:299] Negotiation successful
0620 02:55:16.792488 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11957,"thread_start_us":11788,"threads_started":1}
I20260620 02:55:16.795590 13565 tablet_service.cc:1467] Processing CreateTablet for tablet 76f46efad6f74ea1a2d13a816f89b568 (DEFAULT_TABLE table=test-table [id=a98d25dc02b74a2a8c41a23cf4650ab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:55:16.795840 13700 tablet_service.cc:1467] Processing CreateTablet for tablet 76f46efad6f74ea1a2d13a816f89b568 (DEFAULT_TABLE table=test-table [id=a98d25dc02b74a2a8c41a23cf4650ab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:55:16.796649 13420 tablet_service.cc:1467] Processing CreateTablet for tablet 76f46efad6f74ea1a2d13a816f89b568 (DEFAULT_TABLE table=test-table [id=a98d25dc02b74a2a8c41a23cf4650ab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:55:16.797206 13700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76f46efad6f74ea1a2d13a816f89b568. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:55:16.797263 13565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76f46efad6f74ea1a2d13a816f89b568. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:55:16.797978 13420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76f46efad6f74ea1a2d13a816f89b568. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:55:16.809916 13793 tablet_bootstrap.cc:492] T 76f46efad6f74ea1a2d13a816f89b568 P 3ed297b3acbc41619e78947a13848a00: Bootstrap starting.
I20260620 02:55:16.812979 13795 tablet_bootstrap.cc:492] T 76f46efad6f74ea1a2d13a816f89b568 P ce9df07e95fd4579bbc110f584cd4abf: Bootstrap starting.
I20260620 02:55:16.813779 13794 tablet_bootstrap.cc:492] T 76f46efad6f74ea1a2d13a816f89b568 P 88f69f8286aa473da9efc90ecb187cbe: Bootstrap starting.
I20260620 02:55:16.814106 13793 tablet_bootstrap.cc:654] T 76f46efad6f74ea1a2d13a816f89b568 P 3ed297b3acbc41619e78947a13848a00: Neither blocks nor log segments found. Creating new log.
I20260620 02:55:16.815559 13793 log.cc:824] T 76f46efad6f74ea1a2d13a816f89b568 P 3ed297b3acbc41619e78947a13848a00: Log is configured to *not* fsync() on all Append() calls
I20260620 02:55:16.817641 13794 tablet_bootstrap.cc:654] T 76f46efad6f74ea1a2d13a816f89b568 P 88f69f8286aa473da9efc90ecb187cbe: Neither blocks nor log segments found. Creating new log.
I20260620 02:55:16.818938 13795 tablet_bootstrap.cc:654] T 76f46efad6f74ea1a2d13a816f89b568 P ce9df07e95fd4579bbc110f584cd4abf: Neither blocks nor log segments found. Creating new log.
I20260620 02:55:16.818931 13794 log.cc:824] T 76f46efad6f74ea1a2d13a816f89b568 P 88f69f8286aa473da9efc90ecb187cbe: Log is configured to *not* fsync() on all Append() calls
I20260620 02:55:16.819433 13793 tablet_bootstrap.cc:492] T 76f46efad6f74ea1a2d13a816f89b568 P 3ed297b3acbc41619e78947a13848a00: No bootstrap required, opened a new log
I20260620 02:55:16.819795 13793 ts_tablet_manager.cc:1397] T 76f46efad6f74ea1a2d13a816f89b568 P 3ed297b3acbc41619e78947a13848a00: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260620 02:55:16.820024 13795 log.cc:824] T 76f46efad6f74ea1a2d13a816f89b568 P ce9df07e95fd4579bbc110f584cd4abf: Log is configured to *not* fsync() on all Append() calls
I20260620 02:55:16.823272 13795 tablet_bootstrap.cc:492] T 76f46efad6f74ea1a2d13a816f89b568 P ce9df07e95fd4579bbc110f584cd4abf: No bootstrap required, opened a new log
I20260620 02:55:16.823551 13795 ts_tablet_manager.cc:1397] T 76f46efad6f74ea1a2d13a816f89b568 P ce9df07e95fd4579bbc110f584cd4abf: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.001s
I20260620 02:55:16.823880 13794 tablet_bootstrap.cc:492] T 76f46efad6f74ea1a2d13a816f89b568 P 88f69f8286aa473da9efc90ecb187cbe: No bootstrap required, opened a new log
I20260620 02:55:16.824180 13794 ts_tablet_manager.cc:1397] T 76f46efad6f74ea1a2d13a816f89b568 P 88f69f8286aa473da9efc90ecb187cbe: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260620 02:55:16.830884 13793 raft_consensus.cc:348] T 76f46efad6f74ea1a2d13a816f89b568 P 3ed297b3acbc41619e78947a13848a00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f69f8286aa473da9efc90ecb187cbe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36803 } } peers { permanent_uuid: "ce9df07e95fd4579bbc110f584cd4abf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41733 } } peers { permanent_uuid: "3ed297b3acbc41619e78947a13848a00" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44579 } }
I20260620 02:55:16.831073 13794 raft_consensus.cc:348] T 76f46efad6f74ea1a2d13a816f89b568 P 88f69f8286aa473da9efc90ecb187cbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f69f8286aa473da9efc90ecb187cbe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36803 } } peers { permanent_uuid: "ce9df07e95fd4579bbc110f584cd4abf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41733 } } peers { permanent_uuid: "3ed297b3acbc41619e78947a13848a00" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44579 } }
I20260620 02:55:16.831377 13793 raft_consensus.cc:374] T 76f46efad6f74ea1a2d13a816f89b568 P 3ed297b3acbc41619e78947a13848a00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:55:16.831439 13794 raft_consensus.cc:374] T 76f46efad6f74ea1a2d13a816f89b568 P 88f69f8286aa473da9efc90ecb187cbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:55:16.831503 13793 raft_consensus.cc:729] T 76f46efad6f74ea1a2d13a816f89b568 P 3ed297b3acbc41619e78947a13848a00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ed297b3acbc41619e78947a13848a00, State: Initialized, Role: FOLLOWER
I20260620 02:55:16.831576 13794 raft_consensus.cc:729] T 76f46efad6f74ea1a2d13a816f89b568 P 88f69f8286aa473da9efc90ecb187cbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88f69f8286aa473da9efc90ecb187cbe, State: Initialized, Role: FOLLOWER
I20260620 02:55:16.832250 13794 consensus_queue.cc:260] T 76f46efad6f74ea1a2d13a816f89b568 P 88f69f8286aa473da9efc90ecb187cbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f69f8286aa473da9efc90ecb187cbe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36803 } } peers { permanent_uuid: "ce9df07e95fd4579bbc110f584cd4abf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41733 } } peers { permanent_uuid: "3ed297b3acbc41619e78947a13848a00" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44579 } }
I20260620 02:55:16.832201 13793 consensus_queue.cc:260] T 76f46efad6f74ea1a2d13a816f89b568 P 3ed297b3acbc41619e78947a13848a00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f69f8286aa473da9efc90ecb187cbe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36803 } } peers { permanent_uuid: "ce9df07e95fd4579bbc110f584cd4abf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41733 } } peers { permanent_uuid: "3ed297b3acbc41619e78947a13848a00" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44579 } }
I20260620 02:55:16.833554 13765 heartbeater.cc:503] Master 127.0.97.62:33553 was elected leader, sending a full tablet report...
I20260620 02:55:16.834132 13793 ts_tablet_manager.cc:1428] T 76f46efad6f74ea1a2d13a816f89b568 P 3ed297b3acbc41619e78947a13848a00: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260620 02:55:16.834144 13795 raft_consensus.cc:348] T 76f46efad6f74ea1a2d13a816f89b568 P ce9df07e95fd4579bbc110f584cd4abf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f69f8286aa473da9efc90ecb187cbe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36803 } } peers { permanent_uuid: "ce9df07e95fd4579bbc110f584cd4abf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41733 } } peers { permanent_uuid: "3ed297b3acbc41619e78947a13848a00" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44579 } }
I20260620 02:55:16.834623 13795 raft_consensus.cc:374] T 76f46efad6f74ea1a2d13a816f89b568 P ce9df07e95fd4579bbc110f584cd4abf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:55:16.834746 13795 raft_consensus.cc:729] T 76f46efad6f74ea1a2d13a816f89b568 P ce9df07e95fd4579bbc110f584cd4abf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce9df07e95fd4579bbc110f584cd4abf, State: Initialized, Role: FOLLOWER
I20260620 02:55:16.835901 13486 heartbeater.cc:503] Master 127.0.97.62:33553 was elected leader, sending a full tablet report...
I20260620 02:55:16.836203 13795 consensus_queue.cc:260] T 76f46efad6f74ea1a2d13a816f89b568 P ce9df07e95fd4579bbc110f584cd4abf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f69f8286aa473da9efc90ecb187cbe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36803 } } peers { permanent_uuid: "ce9df07e95fd4579bbc110f584cd4abf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41733 } } peers { permanent_uuid: "3ed297b3acbc41619e78947a13848a00" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44579 } }
I20260620 02:55:16.836824 13794 ts_tablet_manager.cc:1428] T 76f46efad6f74ea1a2d13a816f89b568 P 88f69f8286aa473da9efc90ecb187cbe: Time spent starting tablet: real 0.013s user 0.008s sys 0.003s
I20260620 02:55:16.838461 13795 ts_tablet_manager.cc:1428] T 76f46efad6f74ea1a2d13a816f89b568 P ce9df07e95fd4579bbc110f584cd4abf: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260620 02:55:16.918041 13799 raft_consensus.cc:482] T 76f46efad6f74ea1a2d13a816f89b568 P 88f69f8286aa473da9efc90ecb187cbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:55:16.918483 13799 raft_consensus.cc:504] T 76f46efad6f74ea1a2d13a816f89b568 P 88f69f8286aa473da9efc90ecb187cbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f69f8286aa473da9efc90ecb187cbe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36803 } } peers { permanent_uuid: "ce9df07e95fd4579bbc110f584cd4abf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41733 } } peers { permanent_uuid: "3ed297b3acbc41619e78947a13848a00" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44579 } }
I20260620 02:55:16.920799 13799 leader_election.cc:288] T 76f46efad6f74ea1a2d13a816f89b568 P 88f69f8286aa473da9efc90ecb187cbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce9df07e95fd4579bbc110f584cd4abf (127.0.97.2:41733), 3ed297b3acbc41619e78947a13848a00 (127.0.97.3:44579)
I20260620 02:55:16.928308 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:16.921031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41733 (local address 127.0.97.1:46789)
0620 02:55:16.921705 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:16.921721 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:16.921903 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:16.922496 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:16.922502 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:16.922525 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:55:16.922916 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:16.922929 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:16.924377 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:16.924382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:16.926055 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:16.926069 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:16.927481 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:16.927488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:16.927679 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:16.927715 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 02:55:16.927835 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 02:55:16.927976 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":396,"threads_started":1}
I20260620 02:55:16.928300 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:16.921188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46789 (local address 127.0.97.2:41733)
0620 02:55:16.921475 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:16.921484 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:16.921817 (+ 333us) server_negotiation.cc:407] Connection header received
0620 02:55:16.922046 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:16.922051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:16.922152 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:16.922270 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:55:16.923119 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:16.924233 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:16.926224 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:16.927270 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:16.927369 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:16.927445 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:16.927886 (+ 441us) server_negotiation.cc:299] Negotiation successful
0620 02:55:16.928053 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 02:55:16.928890 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:16.922128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52473 (local address 127.0.97.3:44579)
0620 02:55:16.922337 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:16.922343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:16.922854 (+ 511us) server_negotiation.cc:407] Connection header received
0620 02:55:16.923066 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:16.923071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:16.923150 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:16.923217 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:55:16.923885 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:16.924850 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:16.926842 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:16.927874 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:16.927991 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:16.928085 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:16.928554 (+ 469us) server_negotiation.cc:299] Negotiation successful
0620 02:55:16.928699 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 02:55:16.929497 13803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:16.921847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44579 (local address 127.0.97.1:52473)
0620 02:55:16.922765 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:16.922781 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:16.922904 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:16.923265 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:16.923270 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:16.923298 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:55:16.923691 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:16.923703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:16.925048 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:16.925052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:16.926684 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:16.926698 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:16.928141 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:16.928149 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:16.928342 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:16.928375 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 02:55:16.929126 (+ 751us) client_negotiation.cc:240] Negotiation successful
0620 02:55:16.929232 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"mutex_wait_us":1,"thread_start_us":560,"threads_started":1}
I20260620 02:55:16.929370 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76f46efad6f74ea1a2d13a816f89b568" candidate_uuid: "88f69f8286aa473da9efc90ecb187cbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9df07e95fd4579bbc110f584cd4abf" is_pre_election: true
I20260620 02:55:16.929925 13585 raft_consensus.cc:2454] T 76f46efad6f74ea1a2d13a816f89b568 P ce9df07e95fd4579bbc110f584cd4abf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88f69f8286aa473da9efc90ecb187cbe in term 0.
I20260620 02:55:16.930505 13720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76f46efad6f74ea1a2d13a816f89b568" candidate_uuid: "88f69f8286aa473da9efc90ecb187cbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ed297b3acbc41619e78947a13848a00" is_pre_election: true
I20260620 02:55:16.930644 13375 leader_election.cc:302] T 76f46efad6f74ea1a2d13a816f89b568 P 88f69f8286aa473da9efc90ecb187cbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88f69f8286aa473da9efc90ecb187cbe, ce9df07e95fd4579bbc110f584cd4abf; no voters:
I20260620 02:55:16.931070 13799 raft_consensus.cc:2781] T 76f46efad6f74ea1a2d13a816f89b568 P 88f69f8286aa473da9efc90ecb187cbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:55:16.931062 13720 raft_consensus.cc:2454] T 76f46efad6f74ea1a2d13a816f89b568 P 3ed297b3acbc41619e78947a13848a00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88f69f8286aa473da9efc90ecb187cbe in term 0.
I20260620 02:55:16.931247 13799 raft_consensus.cc:482] T 76f46efad6f74ea1a2d13a816f89b568 P 88f69f8286aa473da9efc90ecb187cbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:55:16.931365 13799 raft_consensus.cc:3037] T 76f46efad6f74ea1a2d13a816f89b568 P 88f69f8286aa473da9efc90ecb187cbe [term 0 FOLLOWER]: Advancing to term 1
W20260620 02:55:16.931507 13766 tablet.cc:2369] T 76f46efad6f74ea1a2d13a816f89b568 P 3ed297b3acbc41619e78947a13848a00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:55:16.933812 13799 raft_consensus.cc:504] T 76f46efad6f74ea1a2d13a816f89b568 P 88f69f8286aa473da9efc90ecb187cbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f69f8286aa473da9efc90ecb187cbe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36803 } } peers { permanent_uuid: "ce9df07e95fd4579bbc110f584cd4abf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41733 } } peers { permanent_uuid: "3ed297b3acbc41619e78947a13848a00" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44579 } }
I20260620 02:55:16.934443 13799 leader_election.cc:288] T 76f46efad6f74ea1a2d13a816f89b568 P 88f69f8286aa473da9efc90ecb187cbe [CANDIDATE]: Term 1 election: Requested vote from peers ce9df07e95fd4579bbc110f584cd4abf (127.0.97.2:41733), 3ed297b3acbc41619e78947a13848a00 (127.0.97.3:44579)
I20260620 02:55:16.935061 13585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76f46efad6f74ea1a2d13a816f89b568" candidate_uuid: "88f69f8286aa473da9efc90ecb187cbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9df07e95fd4579bbc110f584cd4abf"
I20260620 02:55:16.935133 13720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76f46efad6f74ea1a2d13a816f89b568" candidate_uuid: "88f69f8286aa473da9efc90ecb187cbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ed297b3acbc41619e78947a13848a00"
I20260620 02:55:16.935266 13585 raft_consensus.cc:3037] T 76f46efad6f74ea1a2d13a816f89b568 P ce9df07e95fd4579bbc110f584cd4abf [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:55:16.935379 13720 raft_consensus.cc:3037] T 76f46efad6f74ea1a2d13a816f89b568 P 3ed297b3acbc41619e78947a13848a00 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:55:16.937539 13720 raft_consensus.cc:2454] T 76f46efad6f74ea1a2d13a816f89b568 P 3ed297b3acbc41619e78947a13848a00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88f69f8286aa473da9efc90ecb187cbe in term 1.
I20260620 02:55:16.937809 13585 raft_consensus.cc:2454] T 76f46efad6f74ea1a2d13a816f89b568 P ce9df07e95fd4579bbc110f584cd4abf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88f69f8286aa473da9efc90ecb187cbe in term 1.
I20260620 02:55:16.937940 13375 leader_election.cc:302] T 76f46efad6f74ea1a2d13a816f89b568 P 88f69f8286aa473da9efc90ecb187cbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed297b3acbc41619e78947a13848a00, 88f69f8286aa473da9efc90ecb187cbe; no voters:
I20260620 02:55:16.938321 13799 raft_consensus.cc:2781] T 76f46efad6f74ea1a2d13a816f89b568 P 88f69f8286aa473da9efc90ecb187cbe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:55:16.938834 13799 raft_consensus.cc:686] T 76f46efad6f74ea1a2d13a816f89b568 P 88f69f8286aa473da9efc90ecb187cbe [term 1 LEADER]: Becoming Leader. State: Replica: 88f69f8286aa473da9efc90ecb187cbe, State: Running, Role: LEADER
I20260620 02:55:16.939435 13799 consensus_queue.cc:237] T 76f46efad6f74ea1a2d13a816f89b568 P 88f69f8286aa473da9efc90ecb187cbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f69f8286aa473da9efc90ecb187cbe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36803 } } peers { permanent_uuid: "ce9df07e95fd4579bbc110f584cd4abf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41733 } } peers { permanent_uuid: "3ed297b3acbc41619e78947a13848a00" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44579 } }
W20260620 02:55:16.943627 13631 tablet.cc:2369] T 76f46efad6f74ea1a2d13a816f89b568 P ce9df07e95fd4579bbc110f584cd4abf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:55:16.944178 13286 catalog_manager.cc:5495] T 76f46efad6f74ea1a2d13a816f89b568 P 88f69f8286aa473da9efc90ecb187cbe reported cstate change: term changed from 0 to 1, leader changed from <none> to 88f69f8286aa473da9efc90ecb187cbe (127.0.97.1). New cstate: current_term: 1 leader_uuid: "88f69f8286aa473da9efc90ecb187cbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f69f8286aa473da9efc90ecb187cbe" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce9df07e95fd4579bbc110f584cd4abf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ed297b3acbc41619e78947a13848a00" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44579 } health_report { overall_health: UNKNOWN } } }
I20260620 02:55:16.990468 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:16.984568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39344 (local address 127.0.97.1:36803)
0620 02:55:16.984797 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:16.984802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:16.984966 (+ 164us) server_negotiation.cc:407] Connection header received
0620 02:55:16.985206 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:16.985212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:16.985315 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:16.985452 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:55:16.985881 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:16.986794 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:16.988236 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:16.988662 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:16.988772 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:16.989335 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:55:16.989873 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:55:16.989981 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:16.990133 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 02:55:16.990208 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 02:55:16.999215 13720 raft_consensus.cc:1261] T 76f46efad6f74ea1a2d13a816f89b568 P 3ed297b3acbc41619e78947a13848a00 [term 1 FOLLOWER]: Refusing update from remote peer 88f69f8286aa473da9efc90ecb187cbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:55:16.999300 13585 raft_consensus.cc:1261] T 76f46efad6f74ea1a2d13a816f89b568 P ce9df07e95fd4579bbc110f584cd4abf [term 1 FOLLOWER]: Refusing update from remote peer 88f69f8286aa473da9efc90ecb187cbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:55:17.000198 13804 consensus_queue.cc:1035] T 76f46efad6f74ea1a2d13a816f89b568 P 88f69f8286aa473da9efc90ecb187cbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9df07e95fd4579bbc110f584cd4abf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:55:17.000627 13799 consensus_queue.cc:1035] T 76f46efad6f74ea1a2d13a816f89b568 P 88f69f8286aa473da9efc90ecb187cbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed297b3acbc41619e78947a13848a00" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:55:17.012740 13807 mvcc.cc:205] Tried to move back new op lower bound from 7298761183218438144 to 7298761182988308480. Current Snapshot: MvccSnapshot[applied={T|T < 7298761183218438144}]
I20260620 02:55:17.030479 13809 mvcc.cc:205] Tried to move back new op lower bound from 7298761183218438144 to 7298761182988308480. Current Snapshot: MvccSnapshot[applied={T|T < 7298761183218438144}]
I20260620 02:55:17.034991 13286 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:55:17.035321 13286 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:55:17.039358 13286 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9bf06e47566243ad8538f7900efbbe49: Sending DeleteTablet for 3 replicas of tablet 76f46efad6f74ea1a2d13a816f89b568
I20260620 02:55:17.040509 13565 tablet_service.cc:1514] Processing DeleteTablet for tablet 76f46efad6f74ea1a2d13a816f89b568 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:55:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39060
I20260620 02:55:17.040509 13420 tablet_service.cc:1514] Processing DeleteTablet for tablet 76f46efad6f74ea1a2d13a816f89b568 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:55:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39338
I20260620 02:55:17.040874 13700 tablet_service.cc:1514] Processing DeleteTablet for tablet 76f46efad6f74ea1a2d13a816f89b568 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:55:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52176
I20260620 02:55:17.047825 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 13325
I20260620 02:55:17.052882 13819 tablet_replica.cc:330] T 76f46efad6f74ea1a2d13a816f89b568 P 3ed297b3acbc41619e78947a13848a00: stopping tablet replica
I20260620 02:55:17.053390 13819 raft_consensus.cc:2229] T 76f46efad6f74ea1a2d13a816f89b568 P 3ed297b3acbc41619e78947a13848a00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:55:17.053970 13819 raft_consensus.cc:2258] T 76f46efad6f74ea1a2d13a816f89b568 P 3ed297b3acbc41619e78947a13848a00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:55:17.060030 13819 ts_tablet_manager.cc:1905] T 76f46efad6f74ea1a2d13a816f89b568 P 3ed297b3acbc41619e78947a13848a00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:55:17.066608 13818 tablet_replica.cc:330] T 76f46efad6f74ea1a2d13a816f89b568 P 88f69f8286aa473da9efc90ecb187cbe: stopping tablet replica
I20260620 02:55:17.067528 13818 raft_consensus.cc:2229] T 76f46efad6f74ea1a2d13a816f89b568 P 88f69f8286aa473da9efc90ecb187cbe [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:55:17.068342 13818 raft_consensus.cc:2258] T 76f46efad6f74ea1a2d13a816f89b568 P 88f69f8286aa473da9efc90ecb187cbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:55:17.069185 13817 tablet_replica.cc:330] T 76f46efad6f74ea1a2d13a816f89b568 P ce9df07e95fd4579bbc110f584cd4abf: stopping tablet replica
I20260620 02:55:17.069679 13817 raft_consensus.cc:2229] T 76f46efad6f74ea1a2d13a816f89b568 P ce9df07e95fd4579bbc110f584cd4abf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:55:17.070998 13819 ts_tablet_manager.cc:1918] T 76f46efad6f74ea1a2d13a816f89b568 P 3ed297b3acbc41619e78947a13848a00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:55:17.071070 13817 raft_consensus.cc:2258] T 76f46efad6f74ea1a2d13a816f89b568 P ce9df07e95fd4579bbc110f584cd4abf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:55:17.071170 13819 log.cc:1196] T 76f46efad6f74ea1a2d13a816f89b568 P 3ed297b3acbc41619e78947a13848a00: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-2/wal/wals/76f46efad6f74ea1a2d13a816f89b568
I20260620 02:55:17.071892 13819 ts_tablet_manager.cc:1939] T 76f46efad6f74ea1a2d13a816f89b568 P 3ed297b3acbc41619e78947a13848a00: Deleting consensus metadata
I20260620 02:55:17.071943 13818 ts_tablet_manager.cc:1905] T 76f46efad6f74ea1a2d13a816f89b568 P 88f69f8286aa473da9efc90ecb187cbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:55:17.074841 13817 ts_tablet_manager.cc:1905] T 76f46efad6f74ea1a2d13a816f89b568 P ce9df07e95fd4579bbc110f584cd4abf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:55:17.076001 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:17.052133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39350 (local address 127.0.97.1:36803)
0620 02:55:17.052351 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:17.052359 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:17.052380 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:55:17.052451 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:17.052457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:17.052543 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:17.052638 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:17.054576 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:17.056194 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:17.060467 (+ 4273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:17.061160 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:17.061295 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:17.064878 (+ 3583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:17.064891 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:17.064913 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:17.064956 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:17.069884 (+ 4928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:17.071659 (+ 1775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:17.071669 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:17.071673 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:17.071839 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:17.073050 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:17.073059 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:17.073062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:17.073533 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:17.075172 (+ 1639us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:17.075326 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 02:55:17.075591 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 02:55:17.077000 13481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:55:17.078042 13274 catalog_manager.cc:4841] TS 3ed297b3acbc41619e78947a13848a00 (127.0.97.3:44579): tablet 76f46efad6f74ea1a2d13a816f89b568 (table test-table [id=a98d25dc02b74a2a8c41a23cf4650ab2]) successfully deleted
I20260620 02:55:17.082393 13817 ts_tablet_manager.cc:1918] T 76f46efad6f74ea1a2d13a816f89b568 P ce9df07e95fd4579bbc110f584cd4abf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:55:17.082554 13817 log.cc:1196] T 76f46efad6f74ea1a2d13a816f89b568 P ce9df07e95fd4579bbc110f584cd4abf: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781923914418765-388-0/minicluster-data/ts-1/wal/wals/76f46efad6f74ea1a2d13a816f89b568
I20260620 02:55:17.083055 13817 ts_tablet_manager.cc:1939] T 76f46efad6f74ea1a2d13a816f89b568 P ce9df07e95fd4579bbc110f584cd4abf: Deleting consensus metadata
I20260620 02:55:17.084877 13274 catalog_manager.cc:4841] TS ce9df07e95fd4579bbc110f584cd4abf (127.0.97.2:41733): tablet 76f46efad6f74ea1a2d13a816f89b568 (table test-table [id=a98d25dc02b74a2a8c41a23cf4650ab2]) successfully deleted
I20260620 02:55:17.178345 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 13325
I20260620 02:55:17.196973 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 13490
W20260620 02:55:17.196856 13275 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:36803 (error 108)
I20260620 02:55:17.199322 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:17.198782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36803 (local address 127.0.0.1:39362)
0620 02:55:17.198978 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:17.199130 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 02:55:17.199618 13275 catalog_manager.cc:4584] TS 88f69f8286aa473da9efc90ecb187cbe (127.0.97.1:36803): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 76f46efad6f74ea1a2d13a816f89b568: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36803: connect: Connection refused (error 111)
I20260620 02:55:17.211645 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:17.197922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39076 (local address 127.0.97.2:41733)
0620 02:55:17.198118 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:17.198125 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:17.198148 (+ 23us) server_negotiation.cc:407] Connection header received
0620 02:55:17.198398 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:17.198405 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:17.198506 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:17.198643 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:17.200236 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:17.201006 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:17.202310 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:17.202812 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:17.202923 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:17.205734 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:17.205749 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:17.205764 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:17.205798 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:17.208937 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:17.209609 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:17.209616 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:17.209618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:17.209751 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:17.210099 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:17.210102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:17.210104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:17.210530 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:17.210754 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:17.211057 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 02:55:17.211258 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 02:55:17.212489 13625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:55:17.253638 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:17.253063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36803 (local address 127.0.0.1:39366)
0620 02:55:17.253362 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:17.253471 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260620 02:55:17.254858 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:17.254482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36803 (local address 127.0.0.1:39380)
0620 02:55:17.254640 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:17.254741 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 02:55:17.304365 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 13490
I20260620 02:55:17.315636 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:17.315169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36803 (local address 127.0.0.1:39390)
0620 02:55:17.315392 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:17.315483 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 02:55:17.316725 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:17.316378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36803 (local address 127.0.0.1:39396)
0620 02:55:17.316533 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:17.316619 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 02:55:17.321128 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 13634
I20260620 02:55:17.334334 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:17.321852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52178 (local address 127.0.97.3:44579)
0620 02:55:17.322054 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:17.322059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:17.322077 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:55:17.322167 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:17.322170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:17.322233 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:17.322323 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:17.323622 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:17.324481 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:17.325771 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:17.326220 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:17.326323 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:17.328911 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:17.328925 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:17.328941 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:17.328973 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:17.331890 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:17.332522 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:17.332531 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:17.332533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:17.332651 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:17.332952 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:17.332956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:17.332958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:17.333336 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:17.333560 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:17.333796 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 02:55:17.333955 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 02:55:17.335099 13760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:55:17.397221 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:17.396658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36803 (local address 127.0.0.1:39402)
0620 02:55:17.396904 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:17.397015 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 02:55:17.398389 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:17.397943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36803 (local address 127.0.0.1:39406)
0620 02:55:17.398160 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:17.398240 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 02:55:17.426760 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 13634
I20260620 02:55:17.443678 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 13255
I20260620 02:55:17.445001 13318 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:55:17.561091 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 13255
I20260620 02:55:17.587173 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 02:55:17.923059 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 02:55:17.923298 388 mini_ranger.cc:66] Stopping Ranger...
I20260620 02:55:18.269539 388 mini_ranger.cc:68] Stopped Ranger
2026-06-20 02:55:18.269 UTC [12307] LOG: received smart shutdown request
2026-06-20 02:55:18.273 UTC [12307] LOG: background worker "logical replication launcher" (PID 12316) exited with exit code 1
2026-06-20 02:55:18.273 UTC [12311] LOG: shutting down
2026-06-20 02:55:18.483 UTC [12307] LOG: database system is shut down
2026-06-20T02:55:18Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (55364 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:55:18 dist-test-slave-8p8c krb5kdc[13833](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:55:18 dist-test-slave-8p8c krb5kdc[13833](info): set up 2 sockets
Jun 20 02:55:18 dist-test-slave-8p8c krb5kdc[13833](info): commencing operation
krb5kdc: starting...
W20260620 02:55:19.738883 388 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.008s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:55:19 dist-test-slave-8p8c krb5kdc[13833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:55:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:55:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:55:19.984980 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38529
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38529,127.0.97.61:38517,127.0.97.60:36811
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:55:20.318662 13846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:55:20.319023 13846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:55:20.319190 13846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:55:20.326340 13846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:55:20.326510 13846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:55:20.326622 13846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:55:20.326751 13846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:55:20.326838 13846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:55:20.336701 13846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38529,127.0.97.61:38517,127.0.97.60:36811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:55:20.338121 13846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:55:20.339504 13846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:55:20.351805 13851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:20.351728 13854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:20.351827 13852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:55:20.352576 13846 server_base.cc:1034] running on GCE node
I20260620 02:55:20.353400 13846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:55:20.354905 13846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:55:20.356109 13846 hybrid_clock.cc:648] HybridClock initialized: now 1781924120356068 us; error 35 us; skew 500 ppm
I20260620 02:55:20.356480 13846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:55:20 dist-test-slave-8p8c krb5kdc[13833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:55:20.360036 13846 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:55:20.361675 13846 webserver.cc:460] Webserver started at http://127.0.97.62:41395/ using document root <none> and password file <none>
I20260620 02:55:20.362449 13846 fs_manager.cc:359] Metadata directory not provided
I20260620 02:55:20.362601 13846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:55:20.362906 13846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:55:20.365949 13846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "5dad7eeb782c49e09fcf2528c14c6162"
format_stamp: "Formatted at 2026-06-20 02:55:20 on dist-test-slave-8p8c"
server_key: "50a8b3b9dab9cf3a68418d0de41b6825"
server_key_iv: "debe87d1c7890d72335d0f21a4ee6f2e"
server_key_version: "encryptionkey@0"
I20260620 02:55:20.366814 13846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "5dad7eeb782c49e09fcf2528c14c6162"
format_stamp: "Formatted at 2026-06-20 02:55:20 on dist-test-slave-8p8c"
server_key: "50a8b3b9dab9cf3a68418d0de41b6825"
server_key_iv: "debe87d1c7890d72335d0f21a4ee6f2e"
server_key_version: "encryptionkey@0"
I20260620 02:55:20.372216 13846 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 02:55:20.375653 13861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:20.377084 13846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:55:20.377279 13846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "5dad7eeb782c49e09fcf2528c14c6162"
format_stamp: "Formatted at 2026-06-20 02:55:20 on dist-test-slave-8p8c"
server_key: "50a8b3b9dab9cf3a68418d0de41b6825"
server_key_iv: "debe87d1c7890d72335d0f21a4ee6f2e"
server_key_version: "encryptionkey@0"
I20260620 02:55:20.377492 13846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:55:20.400482 13846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:55:20.405081 13846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:55:20.405367 13846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:55:20.424540 13846 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38529
I20260620 02:55:20.424594 13912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38529 every 8 connection(s)
I20260620 02:55:20.426007 13846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:55:20.429195 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 13846
I20260620 02:55:20.429965 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:55:20.430853 388 external_mini_cluster.cc:1428] Setting key 7a829993f093e510426ba727ce31420f
I20260620 02:55:20.433430 13913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:55:20.440752 13913 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38529 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Jun 20 02:55:20 dist-test-slave-8p8c krb5kdc[13833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:55:20.471932 13916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:20.444411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40114 (local address 127.0.97.62:38529)
0620 02:55:20.445468 (+ 1057us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:20.445486 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:20.446097 (+ 611us) server_negotiation.cc:407] Connection header received
0620 02:55:20.449049 (+ 2952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:20.449060 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:20.449448 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:20.449721 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:20.451113 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:20.453085 (+ 1972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:20.454950 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:20.455961 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:20.456124 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:20.461059 (+ 4935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:20.461074 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:20.461097 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:20.461150 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:20.464707 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:20.465938 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:20.465988 (+ 50us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:20.465991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:20.466175 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:20.466876 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:20.466885 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:20.466887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:20.467527 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:20.467879 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:20.469233 (+ 1354us) server_negotiation.cc:299] Negotiation successful
0620 02:55:20.469829 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":255,"server-negotiator.queue_time_us":788,"thread_start_us":529,"threads_started":1}
I20260620 02:55:20.471992 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:20.444411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38529 (local address 127.0.0.1:40114)
0620 02:55:20.445880 (+ 1469us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:20.445928 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:20.448540 (+ 2612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:20.449812 (+ 1272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:20.449821 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:20.450529 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:55:20.450888 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:20.450908 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:20.453394 (+ 2486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:20.453403 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:20.454780 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:20.454796 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:20.456272 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:20.456283 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:20.456521 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:20.457327 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:55:20.457371 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:55:20.460741 (+ 3370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:55:20.464880 (+ 4139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:20.464898 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:20.464915 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:20.465708 (+ 793us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:20.466319 (+ 611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:20.466327 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:20.466334 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:20.466630 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:20.467804 (+ 1174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:55:20.467811 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:55:20.468251 (+ 440us) client_negotiation.cc:769] Sending connection context
0620 02:55:20.468747 (+ 496us) client_negotiation.cc:240] Negotiation successful
0620 02:55:20.469539 (+ 792us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1198,"spinlock_wait_cycles":59648,"thread_start_us":935,"threads_started":1}
I20260620 02:55:20.476292 13913 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38517 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:55:20.477841 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:20.477188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38517 (local address 127.0.0.1:55250)
0620 02:55:20.477387 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:20.477715 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260620 02:55:20.478334 13862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38517: connect: Connection refused (error 111)
I20260620 02:55:20.480316 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:20.479787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38517 (local address 127.0.0.1:55256)
0620 02:55:20.479939 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:20.480032 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 02:55:20.480532 13913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38517: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38517: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260620 02:55:20.495429 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38517
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38529,127.0.97.61:38517,127.0.97.60:36811
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:55:20.529891 13913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38517 } attempt: 1
I20260620 02:55:20.531186 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:20.530735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38517 (local address 127.0.0.1:55260)
0620 02:55:20.530937 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:20.531035 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 02:55:20.532451 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:20.532035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38517 (local address 127.0.0.1:55272)
0620 02:55:20.532210 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:20.532314 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 02:55:20.532637 13913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38517: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38517: connect: Connection refused (error 111)
I20260620 02:55:20.600911 13913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38517 } attempt: 2
I20260620 02:55:20.602283 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:20.601684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38517 (local address 127.0.0.1:55284)
0620 02:55:20.601978 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:20.602099 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 02:55:20.603701 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:20.603148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38517 (local address 127.0.0.1:55296)
0620 02:55:20.603360 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:20.603488 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260620 02:55:20.603919 13913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38517: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38517: connect: Connection refused (error 111)
I20260620 02:55:20.695240 13913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38517 } attempt: 3
I20260620 02:55:20.696519 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:20.696022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38517 (local address 127.0.0.1:55306)
0620 02:55:20.696272 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:20.696371 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260620 02:55:20.697706 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:20.697325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38517 (local address 127.0.0.1:55310)
0620 02:55:20.697484 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:20.697560 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 02:55:20.697959 13913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38517: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38517: connect: Connection refused (error 111)
W20260620 02:55:20.839286 13920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:55:20.839663 13920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:55:20.839843 13920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 02:55:20.841217 13913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38517 } attempt: 4
I20260620 02:55:20.842629 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:20.841929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38517 (local address 127.0.0.1:55312)
0620 02:55:20.842273 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:20.842471 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
I20260620 02:55:20.844008 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:20.843555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38517 (local address 127.0.0.1:55320)
0620 02:55:20.843774 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:20.843878 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 02:55:20.844262 13913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38517: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38517: connect: Connection refused (error 111)
W20260620 02:55:20.848521 13920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:55:20.848681 13920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:55:20.848778 13920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:55:20.848881 13920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:55:20.848965 13920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:55:20.859563 13920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38529,127.0.97.61:38517,127.0.97.60:36811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:55:20.861161 13920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:55:20.862639 13920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:55:20.876202 13925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:20.876202 13926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:20.876153 13928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:55:20.876830 13920 server_base.cc:1034] running on GCE node
I20260620 02:55:20.877722 13920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:55:20.879285 13920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:55:20.880661 13920 hybrid_clock.cc:648] HybridClock initialized: now 1781924120880616 us; error 38 us; skew 500 ppm
I20260620 02:55:20.880978 13920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:55:20 dist-test-slave-8p8c krb5kdc[13833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:55:20.888075 13920 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260620 02:55:20.889778 13920 webserver.cc:460] Webserver started at http://127.0.97.61:33401/ using document root <none> and password file <none>
I20260620 02:55:20.890590 13920 fs_manager.cc:359] Metadata directory not provided
I20260620 02:55:20.890730 13920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:55:20.891047 13920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:55:20.894155 13920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-1/data/instance:
uuid: "ed830c43b38a4894b5b920c995c73c0c"
format_stamp: "Formatted at 2026-06-20 02:55:20 on dist-test-slave-8p8c"
server_key: "2d0b0d03fcaa41b330e9924670358dad"
server_key_iv: "1cfd706e6df65dd5ec9ef17068c54feb"
server_key_version: "encryptionkey@0"
I20260620 02:55:20.895141 13920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-1/wal/instance:
uuid: "ed830c43b38a4894b5b920c995c73c0c"
format_stamp: "Formatted at 2026-06-20 02:55:20 on dist-test-slave-8p8c"
server_key: "2d0b0d03fcaa41b330e9924670358dad"
server_key_iv: "1cfd706e6df65dd5ec9ef17068c54feb"
server_key_version: "encryptionkey@0"
I20260620 02:55:20.900573 13920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:55:20.904183 13935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:20.905721 13920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:55:20.905951 13920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-1/wal
uuid: "ed830c43b38a4894b5b920c995c73c0c"
format_stamp: "Formatted at 2026-06-20 02:55:20 on dist-test-slave-8p8c"
server_key: "2d0b0d03fcaa41b330e9924670358dad"
server_key_iv: "1cfd706e6df65dd5ec9ef17068c54feb"
server_key_version: "encryptionkey@0"
I20260620 02:55:20.906174 13920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:55:20.935882 13920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:55:20.941069 13920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:55:20.941386 13920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:55:20.963692 13920 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:38517
I20260620 02:55:20.963737 13986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:38517 every 8 connection(s)
I20260620 02:55:20.965206 13920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-1/data/info.pb
I20260620 02:55:20.969239 13987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:55:20.970742 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 13920
I20260620 02:55:20.970952 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-1/wal/instance
I20260620 02:55:20.971392 388 external_mini_cluster.cc:1428] Setting key 07212729d6806b991ac3b86c5a1fa787
I20260620 02:55:20.976822 13987 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38529 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Jun 20 02:55:20 dist-test-slave-8p8c krb5kdc[13833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:55:21.002974 13990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:20.979752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40130 (local address 127.0.97.62:38529)
0620 02:55:20.980315 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:20.980321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:20.981452 (+ 1131us) server_negotiation.cc:407] Connection header received
0620 02:55:20.983907 (+ 2455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:20.983917 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:20.984017 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:20.984152 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:20.986027 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:20.987435 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:20.989226 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:20.989977 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:20.990142 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:20.996726 (+ 6584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:20.996740 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:20.996745 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:20.996800 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:20.999631 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:21.000471 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:21.000477 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:21.000481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:21.000647 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:21.001106 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:21.001110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:21.001113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:21.001418 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:21.001580 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:21.002332 (+ 752us) server_negotiation.cc:299] Negotiation successful
0620 02:55:21.002586 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":284,"threads_started":1}
I20260620 02:55:21.004034 13991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:20.980358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38529 (local address 127.0.0.1:40130)
0620 02:55:20.981252 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:20.981312 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:20.983594 (+ 2282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:20.984606 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:20.984616 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:20.985184 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:55:20.985811 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:20.985831 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:20.987639 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:20.987644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:20.989047 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:20.989062 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:20.990264 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:20.990270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:20.990589 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:20.991355 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:55:20.991380 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:55:20.996323 (+ 4943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:55:20.999821 (+ 3498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:20.999832 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:20.999852 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:21.000334 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:21.000771 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:21.000776 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:21.000780 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:21.000989 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:21.001553 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:55:21.001561 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:55:21.001947 (+ 386us) client_negotiation.cc:769] Sending connection context
0620 02:55:21.002244 (+ 297us) client_negotiation.cc:240] Negotiation successful
0620 02:55:21.002830 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":383,"threads_started":1}
I20260620 02:55:21.005872 13987 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38517 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:55:21 dist-test-slave-8p8c krb5kdc[13833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 02:55:21.023327 13991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.006702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38517 (local address 127.0.0.1:55334)
0620 02:55:21.006944 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:21.006969 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:21.007113 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:21.008153 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:21.008161 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:21.008713 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:55:21.008927 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:21.008938 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.010768 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.010776 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:21.012393 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:21.012408 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.013395 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.013403 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:21.013698 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:21.014402 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:55:21.014440 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:55:21.016959 (+ 2519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:55:21.020559 (+ 3600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:21.020566 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:21.020571 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:21.020996 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:21.021398 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:21.021406 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:21.021411 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:21.021537 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:21.022212 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:55:21.022217 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:55:21.022426 (+ 209us) client_negotiation.cc:769] Sending connection context
0620 02:55:21.022577 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 02:55:21.022874 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 02:55:21.023407 13993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.006933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55334 (local address 127.0.97.61:38517)
0620 02:55:21.007689 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:21.007704 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:21.007730 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:55:21.007778 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:21.007784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:21.007969 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:21.008097 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:21.009127 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.010617 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.012621 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.013215 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.013325 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:21.017209 (+ 3884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:21.017222 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:21.017236 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:21.017265 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:21.020410 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:21.021142 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:21.021154 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:21.021159 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:21.021291 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:21.021635 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:21.021640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:21.021645 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:21.022076 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:21.022302 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:21.022725 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 02:55:21.022981 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":595,"thread_start_us":431,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260620 02:55:21.026300 13987 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36811 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:55:21.026927 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:36811
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38529,127.0.97.61:38517,127.0.97.60:36811
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:55:21.027736 13991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.027111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36811 (local address 127.0.0.1:55284)
0620 02:55:21.027349 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:21.027490 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260620 02:55:21.028051 13937 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36811: connect: Connection refused (error 111)
I20260620 02:55:21.029886 13991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.029451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36811 (local address 127.0.0.1:55286)
0620 02:55:21.029627 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:21.029733 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 02:55:21.030136 13987 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36811: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36811: connect: Connection refused (error 111)
I20260620 02:55:21.079473 13987 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36811 } attempt: 1
I20260620 02:55:21.080756 13991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.080266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36811 (local address 127.0.0.1:55290)
0620 02:55:21.080501 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:21.080600 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 02:55:21.081929 13991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.081603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36811 (local address 127.0.0.1:55306)
0620 02:55:21.081749 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:21.081827 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 02:55:21.082262 13987 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36811: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36811: connect: Connection refused (error 111)
I20260620 02:55:21.143497 13913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38517 } attempt: 5
I20260620 02:55:21.150534 13987 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36811 } attempt: 2
I20260620 02:55:21.151675 13991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.151190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36811 (local address 127.0.0.1:55308)
0620 02:55:21.151433 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:21.151528 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
Jun 20 02:55:21 dist-test-slave-8p8c krb5kdc[13833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 02:55:21.153175 13991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.152750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36811 (local address 127.0.0.1:55316)
0620 02:55:21.152924 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:21.153018 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 02:55:21.153502 13987 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36811: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36811: connect: Connection refused (error 111)
I20260620 02:55:21.159119 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.144284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38517 (local address 127.0.0.1:55336)
0620 02:55:21.144578 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:21.144600 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:21.144737 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:21.145231 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:21.145236 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:21.145721 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:55:21.145942 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:21.145954 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.147229 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.147235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:21.148122 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:21.148134 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.149196 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.149202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:21.149407 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:21.150093 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:55:21.150112 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:55:21.152835 (+ 2723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:55:21.156598 (+ 3763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:21.156603 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:21.156605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:21.156986 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:21.157459 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:21.157464 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:21.157466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:21.157622 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:21.158274 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:55:21.158278 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:55:21.158485 (+ 207us) client_negotiation.cc:769] Sending connection context
0620 02:55:21.158636 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 02:55:21.158874 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170}
I20260620 02:55:21.159354 13993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.144391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55336 (local address 127.0.97.61:38517)
0620 02:55:21.144578 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:21.144583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:21.144661 (+ 78us) server_negotiation.cc:407] Connection header received
0620 02:55:21.144922 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:21.144927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:21.145059 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:21.145158 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:21.146186 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.147068 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.148270 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.149013 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.149181 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:21.153021 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:21.153027 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:21.153032 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:21.153072 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:21.156426 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:21.157111 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:21.157117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:21.157121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:21.157269 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:21.157756 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:21.157762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:21.157765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:21.158144 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:21.158291 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:21.158769 (+ 478us) server_negotiation.cc:299] Negotiation successful
0620 02:55:21.158987 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 02:55:21.160480 13913 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36811 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:55:21.161587 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.161147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36811 (local address 127.0.0.1:55332)
0620 02:55:21.161325 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:21.161415 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 02:55:21.162719 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.162257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36811 (local address 127.0.0.1:55342)
0620 02:55:21.162457 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:21.162557 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 02:55:21.163024 13913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36811: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36811: connect: Connection refused (error 111)
I20260620 02:55:21.214322 13913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36811 } attempt: 1
I20260620 02:55:21.215888 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.215201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36811 (local address 127.0.0.1:55356)
0620 02:55:21.215489 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:21.215634 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260620 02:55:21.217182 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.216772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36811 (local address 127.0.0.1:55366)
0620 02:55:21.216937 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:21.217047 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 02:55:21.217453 13913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36811: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36811: connect: Connection refused (error 111)
I20260620 02:55:21.244752 13987 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36811 } attempt: 3
I20260620 02:55:21.246007 13991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.245515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36811 (local address 127.0.0.1:55374)
0620 02:55:21.245755 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:21.245853 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 02:55:21.247174 13991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.246749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36811 (local address 127.0.0.1:55388)
0620 02:55:21.246977 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:21.247053 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260620 02:55:21.247398 13987 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36811: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36811: connect: Connection refused (error 111)
I20260620 02:55:21.285689 13913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36811 } attempt: 2
I20260620 02:55:21.286945 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.286314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36811 (local address 127.0.0.1:55390)
0620 02:55:21.286575 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:21.286711 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 02:55:21.288178 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.287747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36811 (local address 127.0.0.1:55396)
0620 02:55:21.287918 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:21.288028 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 02:55:21.288435 13913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36811: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36811: connect: Connection refused (error 111)
W20260620 02:55:21.365500 13995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:55:21.365829 13995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:55:21.365923 13995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:55:21.373461 13995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:55:21.373584 13995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:55:21.373631 13995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:55:21.373692 13995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:55:21.373737 13995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:55:21.383862 13995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38529,127.0.97.61:38517,127.0.97.60:36811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:36811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:55:21.385139 13995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:55:21.386533 13995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 02:55:21.390616 13987 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36811 } attempt: 4
I20260620 02:55:21.391971 13991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.391277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36811 (local address 127.0.0.1:55404)
0620 02:55:21.391546 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:21.391712 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 02:55:21.394564 13991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.393934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36811 (local address 127.0.0.1:55414)
0620 02:55:21.394222 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:21.394336 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260620 02:55:21.394791 13913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36811 } attempt: 3
W20260620 02:55:21.395337 13987 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36811: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36811: connect: Connection refused (error 111)
I20260620 02:55:21.402587 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.395376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36811 (local address 127.0.0.1:55418)
0620 02:55:21.395634 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:21.395740 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260620 02:55:21.403975 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.403478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36811 (local address 127.0.0.1:55422)
0620 02:55:21.403741 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:21.403832 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
W20260620 02:55:21.404616 13913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36811: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36811: connect: Connection refused (error 111)
W20260620 02:55:21.407866 14003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:21.407835 14001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:21.408623 14000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:55:21.409155 13995 server_base.cc:1034] running on GCE node
I20260620 02:55:21.409978 13995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:55:21.411638 13995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:55:21.412897 13995 hybrid_clock.cc:648] HybridClock initialized: now 1781924121412845 us; error 46 us; skew 500 ppm
I20260620 02:55:21.413342 13995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:55:21 dist-test-slave-8p8c krb5kdc[13833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:55:21.416529 13995 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260620 02:55:21.418033 13995 webserver.cc:460] Webserver started at http://127.0.97.60:34647/ using document root <none> and password file <none>
I20260620 02:55:21.418766 13995 fs_manager.cc:359] Metadata directory not provided
I20260620 02:55:21.418862 13995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:55:21.419108 13995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:55:21.421969 13995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-2/data/instance:
uuid: "dfa74d2d3bb843e88392fc5aa1888221"
format_stamp: "Formatted at 2026-06-20 02:55:21 on dist-test-slave-8p8c"
server_key: "96b9e50b622a9a9ebb2ebeed1a8eff58"
server_key_iv: "d8c6c070ebac405be031b61b83f81b45"
server_key_version: "encryptionkey@0"
I20260620 02:55:21.422756 13995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-2/wal/instance:
uuid: "dfa74d2d3bb843e88392fc5aa1888221"
format_stamp: "Formatted at 2026-06-20 02:55:21 on dist-test-slave-8p8c"
server_key: "96b9e50b622a9a9ebb2ebeed1a8eff58"
server_key_iv: "d8c6c070ebac405be031b61b83f81b45"
server_key_version: "encryptionkey@0"
I20260620 02:55:21.428252 13995 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 02:55:21.431986 14010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:21.433466 13995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:55:21.433672 13995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-2/wal
uuid: "dfa74d2d3bb843e88392fc5aa1888221"
format_stamp: "Formatted at 2026-06-20 02:55:21 on dist-test-slave-8p8c"
server_key: "96b9e50b622a9a9ebb2ebeed1a8eff58"
server_key_iv: "d8c6c070ebac405be031b61b83f81b45"
server_key_version: "encryptionkey@0"
I20260620 02:55:21.433887 13995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:55:21.460219 13995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:55:21.465050 13995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:55:21.465381 13995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:55:21.484745 13995 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:36811
I20260620 02:55:21.484767 14061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:36811 every 8 connection(s)
I20260620 02:55:21.486173 13995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-2/data/info.pb
I20260620 02:55:21.490738 14062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:55:21.494463 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 13995
I20260620 02:55:21.494654 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781923914418765-388-0/minicluster-data/master-2/wal/instance
I20260620 02:55:21.495137 388 external_mini_cluster.cc:1428] Setting key bc93cf214800b0b4910494c730a4d572
I20260620 02:55:21.497807 14062 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38529 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:55:21 dist-test-slave-8p8c krb5kdc[13833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 02:55:21 dist-test-slave-8p8c krb5kdc[13833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:55:21.519060 13990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.502054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40134 (local address 127.0.97.62:38529)
0620 02:55:21.502305 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:21.502314 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:21.502928 (+ 614us) server_negotiation.cc:407] Connection header received
0620 02:55:21.503130 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:21.503136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:21.503211 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:21.503295 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:21.504697 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.505825 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.507800 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.508429 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.508572 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:21.512670 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:21.512675 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:21.512678 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:21.512711 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:21.515950 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:21.516807 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:21.516811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:21.516814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:21.516948 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:21.517352 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:21.517356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:21.517358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:21.517760 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:21.517940 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:21.518641 (+ 701us) server_negotiation.cc:299] Negotiation successful
0620 02:55:21.518811 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 02:55:21.520221 14065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.497159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55436 (local address 127.0.97.60:36811)
0620 02:55:21.497981 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:21.497993 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:21.498039 (+ 46us) server_negotiation.cc:407] Connection header received
0620 02:55:21.499745 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:21.499767 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:21.500190 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:21.500505 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:21.502256 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.504429 (+ 2173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.506326 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.507431 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.507622 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:21.512060 (+ 4438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:21.512118 (+ 58us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:21.512136 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:21.512182 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:21.515114 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:21.515968 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:21.515975 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:21.515981 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:21.516275 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:21.516716 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:21.516722 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:21.516727 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:21.517392 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:21.517600 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:21.518204 (+ 604us) server_negotiation.cc:299] Negotiation successful
0620 02:55:21.518732 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":605,"thread_start_us":379,"threads_started":1}
I20260620 02:55:21.520257 14066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.501933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38529 (local address 127.0.0.1:40134)
0620 02:55:21.502776 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:21.502832 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:21.502977 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:21.503389 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:21.503396 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:21.504185 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:55:21.504537 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:21.504550 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.505986 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.505990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:21.507649 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:21.507660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.508663 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.508668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:21.508787 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:21.509411 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:55:21.509490 (+ 79us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:55:21.512452 (+ 2962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:55:21.516080 (+ 3628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:21.516090 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:21.516097 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:21.516618 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:21.517059 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:21.517063 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:21.517066 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:21.517204 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:21.517960 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:55:21.517969 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:55:21.518278 (+ 309us) client_negotiation.cc:769] Sending connection context
0620 02:55:21.518472 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 02:55:21.518853 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":95,"spinlock_wait_cycles":23808,"thread_start_us":400,"threads_started":1}
I20260620 02:55:21.521852 14062 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38517 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:55:21 dist-test-slave-8p8c krb5kdc[13833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 20 02:55:21 dist-test-slave-8p8c krb5kdc[13833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:55:21.535676 14066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.522673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38517 (local address 127.0.0.1:55346)
0620 02:55:21.522932 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:21.522947 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:21.523051 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:21.523433 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:21.523438 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:21.523849 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:55:21.523984 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:21.523992 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.524965 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.524970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:21.525840 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:21.525851 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.526634 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.526639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:21.526862 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:21.527443 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:55:21.527460 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:55:21.529530 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:55:21.533190 (+ 3660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:21.533195 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:21.533199 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:21.533576 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:21.533958 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:21.533963 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:21.533967 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:21.534121 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:21.534683 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:55:21.534687 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:55:21.534853 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 02:55:21.534994 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 02:55:21.535294 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260620 02:55:21.535694 13993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.522677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55346 (local address 127.0.97.61:38517)
0620 02:55:21.522883 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:21.522889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:21.523013 (+ 124us) server_negotiation.cc:407] Connection header received
0620 02:55:21.523165 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:21.523171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:21.523256 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:21.523348 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:21.524163 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.524821 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.525988 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.526493 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.526629 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:21.529734 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:21.529740 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:21.529744 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:21.529781 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:21.533050 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:21.533706 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:21.533718 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:21.533722 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:21.533842 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:21.534232 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:21.534239 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:21.534242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:21.534570 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:21.534731 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:21.535099 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 02:55:21.535305 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 02:55:21.536882 14062 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36811 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:55:21.539161 13990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.523608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40138 (local address 127.0.97.62:38529)
0620 02:55:21.523755 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:21.523762 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:21.523782 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:55:21.523964 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:21.523969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:21.524054 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:21.524155 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:21.525351 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.526068 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.527434 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.527871 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.528003 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:21.532684 (+ 4681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:21.532690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:21.532693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:21.532726 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:21.535464 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:21.536409 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:21.536415 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:21.536418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:21.536657 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:21.537084 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:21.537089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:21.537092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:21.537441 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:21.537787 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:21.538758 (+ 971us) server_negotiation.cc:299] Negotiation successful
0620 02:55:21.538923 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 02:55:21 dist-test-slave-8p8c krb5kdc[13833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 02:55:21 dist-test-slave-8p8c krb5kdc[13833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 02:55:21.551673 14065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.538079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55440 (local address 127.0.97.60:36811)
0620 02:55:21.538260 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:21.538264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:21.538280 (+ 16us) server_negotiation.cc:407] Connection header received
0620 02:55:21.538319 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:21.538322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:21.538431 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:21.538561 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:21.539418 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.540283 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.541994 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.542511 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.542604 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:21.546219 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:21.546223 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:21.546227 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:21.546254 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:21.548959 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:21.549736 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:21.549743 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:21.549748 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:21.549869 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:21.550230 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:21.550235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:21.550238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:21.550589 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:21.550729 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:21.551077 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 02:55:21.551301 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 02:55:21.551792 14066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.537805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36811 (local address 127.0.0.1:55440)
0620 02:55:21.538022 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:21.538041 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:21.538129 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:21.538576 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:21.538581 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:21.538985 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:55:21.539193 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:21.539224 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.540406 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.540411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:21.541833 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:21.541846 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.542654 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.542660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:21.542879 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:21.543472 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:55:21.543489 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:55:21.546036 (+ 2547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:55:21.549104 (+ 3068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:21.549111 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:21.549115 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:21.549617 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:21.549983 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:21.549988 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:21.549991 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:21.550119 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:21.550707 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:55:21.550712 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:55:21.550849 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 02:55:21.550995 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 02:55:21.551435 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":140}
I20260620 02:55:21.557010 13993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.540937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55350 (local address 127.0.97.61:38517)
0620 02:55:21.541097 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:21.541101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:21.541116 (+ 15us) server_negotiation.cc:407] Connection header received
0620 02:55:21.541233 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:21.541236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:21.541287 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:21.541360 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:21.542860 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.543602 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.545301 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.545880 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.545978 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:21.550953 (+ 4975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:21.550959 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:21.550966 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:21.551005 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:21.554231 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:21.554926 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:21.554932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:21.554935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:21.555097 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:21.555474 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:21.555480 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:21.555484 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:21.555880 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:21.556140 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:21.556461 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 02:55:21.556689 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 02:55:21.557730 14062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfa74d2d3bb843e88392fc5aa1888221: Bootstrap starting.
I20260620 02:55:21.561637 14062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfa74d2d3bb843e88392fc5aa1888221: Neither blocks nor log segments found. Creating new log.
I20260620 02:55:21.562745 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:55:21.562883 14062 log.cc:824] T 00000000000000000000000000000000 P dfa74d2d3bb843e88392fc5aa1888221: Log is configured to *not* fsync() on all Append() calls
I20260620 02:55:21.565742 14062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfa74d2d3bb843e88392fc5aa1888221: No bootstrap required, opened a new log
I20260620 02:55:21.576856 14062 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfa74d2d3bb843e88392fc5aa1888221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dad7eeb782c49e09fcf2528c14c6162" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38529 } } peers { permanent_uuid: "ed830c43b38a4894b5b920c995c73c0c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38517 } } peers { permanent_uuid: "dfa74d2d3bb843e88392fc5aa1888221" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36811 } }
I20260620 02:55:21.577343 14062 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfa74d2d3bb843e88392fc5aa1888221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:55:21.577456 14062 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfa74d2d3bb843e88392fc5aa1888221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa74d2d3bb843e88392fc5aa1888221, State: Initialized, Role: FOLLOWER
I20260620 02:55:21.578291 14062 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfa74d2d3bb843e88392fc5aa1888221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dad7eeb782c49e09fcf2528c14c6162" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38529 } } peers { permanent_uuid: "ed830c43b38a4894b5b920c995c73c0c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38517 } } peers { permanent_uuid: "dfa74d2d3bb843e88392fc5aa1888221" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36811 } }
I20260620 02:55:21.582268 13913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36811 } attempt: 4
I20260620 02:55:21.585335 14075 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfa74d2d3bb843e88392fc5aa1888221 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dad7eeb782c49e09fcf2528c14c6162" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38529 } } peers { permanent_uuid: "ed830c43b38a4894b5b920c995c73c0c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38517 } } peers { permanent_uuid: "dfa74d2d3bb843e88392fc5aa1888221" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36811 } } }
I20260620 02:55:21.585709 14075 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfa74d2d3bb843e88392fc5aa1888221 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:55:21.587849 14062 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfa74d2d3bb843e88392fc5aa1888221 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:55:21.588744 13993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.567862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55356 (local address 127.0.97.61:38517)
0620 02:55:21.568033 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:21.568041 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:21.568961 (+ 920us) server_negotiation.cc:407] Connection header received
0620 02:55:21.569508 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:21.569515 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:21.569589 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:21.569676 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:21.572144 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.573305 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.575951 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.576760 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.576861 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:21.581193 (+ 4332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:21.581199 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:21.581204 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:21.581261 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:21.584995 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:21.586061 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:21.586067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:21.586070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:21.586238 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:21.586827 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:21.586833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:21.586837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:21.587207 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:21.587448 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:21.588122 (+ 674us) server_negotiation.cc:299] Negotiation successful
0620 02:55:21.588368 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 02:55:21.597829 14065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.569389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55456 (local address 127.0.97.60:36811)
0620 02:55:21.569624 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:21.569633 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:21.575343 (+ 5710us) server_negotiation.cc:407] Connection header received
0620 02:55:21.575637 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:21.575643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:21.575734 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:21.575804 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:21.578781 (+ 2977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.580212 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.582960 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.583964 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.584085 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:21.587948 (+ 3863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:21.587958 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:21.587966 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:21.588024 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:21.592884 (+ 4860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:21.594083 (+ 1199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:21.594091 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:21.594097 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:21.594313 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:21.594798 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:21.594805 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:21.594811 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:21.595336 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:21.596816 (+ 1480us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:21.596993 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 02:55:21.597294 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 02:55:21.598358 13990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.568186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40144 (local address 127.0.97.62:38529)
0620 02:55:21.568368 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:21.568375 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:21.568646 (+ 271us) server_negotiation.cc:407] Connection header received
0620 02:55:21.568972 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:21.568978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:21.569071 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:21.569166 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:21.572569 (+ 3403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.574026 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.576532 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.577329 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.577429 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:21.581182 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:21.581190 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:21.581194 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:21.581236 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:21.587043 (+ 5807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:21.595042 (+ 7999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:21.595052 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:21.595058 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:21.595285 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:21.595857 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:21.595865 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:21.595871 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:21.596342 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:21.596824 (+ 482us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:21.597502 (+ 678us) server_negotiation.cc:299] Negotiation successful
0620 02:55:21.597852 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 02:55:21 dist-test-slave-8p8c krb5kdc[13833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 02:55:21.617736 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.582827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36811 (local address 127.0.0.1:55464)
0620 02:55:21.583210 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:21.583234 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:21.583340 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:21.601020 (+ 17680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:21.601027 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:21.601445 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:55:21.601582 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:21.601591 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.603047 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.603054 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:21.604164 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:21.604177 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.604891 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.604897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:21.605111 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:21.605838 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:55:21.605861 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:55:21.609181 (+ 3320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:55:21.614767 (+ 5586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:21.614772 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:21.614776 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:21.615198 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:21.615667 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:21.615672 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:21.615677 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:21.615885 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:21.616677 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:55:21.616683 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:55:21.616859 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 02:55:21.617009 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 02:55:21.617315 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230}
W20260620 02:55:21.618021 14087 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dfa74d2d3bb843e88392fc5aa1888221: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:55:21.618148 14087 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:55:21.618924 14076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.583585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55464 (local address 127.0.97.60:36811)
0620 02:55:21.599972 (+ 16387us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:21.599978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:21.599999 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:55:21.600045 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:21.600052 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:21.600865 (+ 813us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:21.600990 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:21.601795 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.602920 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.604309 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.604740 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.604866 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:21.609409 (+ 4543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:21.609415 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:21.609418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:21.609447 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:21.614574 (+ 5127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:21.615376 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:21.615384 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:21.615388 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:21.615544 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:21.615995 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:21.616002 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:21.616005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:21.616450 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:21.616589 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:21.618152 (+ 1563us) server_negotiation.cc:299] Negotiation successful
0620 02:55:21.618441 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16230,"thread_start_us":17129,"threads_started":1}
I20260620 02:55:21.625331 13913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162: Bootstrap starting.
I20260620 02:55:21.616051 13995 master_runner.cc:186] Error getting master registration for 127.0.97.62:38529: OK, instance_id { permanent_uuid: "5dad7eeb782c49e09fcf2528c14c6162" instance_seqno: 1781924120419301 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:55:21.626847 13995 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:55:21.628857 13913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162: Neither blocks nor log segments found. Creating new log.
I20260620 02:55:21.629902 13913 log.cc:824] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162: Log is configured to *not* fsync() on all Append() calls
I20260620 02:55:21.633101 13913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162: No bootstrap required, opened a new log
I20260620 02:55:21.640723 13913 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dad7eeb782c49e09fcf2528c14c6162" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38529 } } peers { permanent_uuid: "ed830c43b38a4894b5b920c995c73c0c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38517 } } peers { permanent_uuid: "dfa74d2d3bb843e88392fc5aa1888221" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36811 } }
I20260620 02:55:21.641137 13913 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:55:21.641274 13913 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dad7eeb782c49e09fcf2528c14c6162, State: Initialized, Role: FOLLOWER
I20260620 02:55:21.642088 13913 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dad7eeb782c49e09fcf2528c14c6162" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38529 } } peers { permanent_uuid: "ed830c43b38a4894b5b920c995c73c0c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38517 } } peers { permanent_uuid: "dfa74d2d3bb843e88392fc5aa1888221" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36811 } }
I20260620 02:55:21.643128 14090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dad7eeb782c49e09fcf2528c14c6162" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38529 } } peers { permanent_uuid: "ed830c43b38a4894b5b920c995c73c0c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38517 } } peers { permanent_uuid: "dfa74d2d3bb843e88392fc5aa1888221" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36811 } } }
I20260620 02:55:21.643565 14090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:55:21.644142 13913 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 02:55:21.655304 14101 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:55:21.655478 14101 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 02:55:21.658293 13867 tablet.cc:2369] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:55:21.654170 13846 master_runner.cc:186] Error getting master registration for 127.0.97.61:38517: OK, instance_id { permanent_uuid: "ed830c43b38a4894b5b920c995c73c0c" instance_seqno: 1781924120956760 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:55:21.663743 13846 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:55:21.694576 13987 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36811 } attempt: 5
Jun 20 02:55:21 dist-test-slave-8p8c krb5kdc[13833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 02:55:21.704627 14090 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:55:21.704986 14090 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dad7eeb782c49e09fcf2528c14c6162" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38529 } } peers { permanent_uuid: "ed830c43b38a4894b5b920c995c73c0c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38517 } } peers { permanent_uuid: "dfa74d2d3bb843e88392fc5aa1888221" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36811 } }
I20260620 02:55:21.706863 14090 leader_election.cc:288] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed830c43b38a4894b5b920c995c73c0c (127.0.97.61:38517), dfa74d2d3bb843e88392fc5aa1888221 (127.0.97.60:36811)
I20260620 02:55:21.707978 14037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5dad7eeb782c49e09fcf2528c14c6162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa74d2d3bb843e88392fc5aa1888221" is_pre_election: true
I20260620 02:55:21.708668 14037 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dfa74d2d3bb843e88392fc5aa1888221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dad7eeb782c49e09fcf2528c14c6162 in term 0.
I20260620 02:55:21.709337 13863 leader_election.cc:302] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dad7eeb782c49e09fcf2528c14c6162, dfa74d2d3bb843e88392fc5aa1888221; no voters:
I20260620 02:55:21.710071 13991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.695189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36811 (local address 127.0.0.1:55470)
0620 02:55:21.695447 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:21.695465 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:21.695594 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:21.695990 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:21.695997 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:21.696432 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:55:21.696601 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:21.696610 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.697724 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.697731 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:21.698719 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:21.698731 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.699463 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.699469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:21.699665 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:21.700289 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:55:21.700307 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:55:21.702439 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:55:21.706789 (+ 4350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:21.706795 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:21.706802 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:21.707408 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:21.707865 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:21.707869 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:21.707873 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:21.708059 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:21.708943 (+ 884us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:55:21.708947 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:55:21.709135 (+ 188us) client_negotiation.cc:769] Sending connection context
0620 02:55:21.709278 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 02:55:21.709675 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260620 02:55:21.710515 14090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:55:21.710693 14090 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:55:21.710813 14090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:55:21.711545 14076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:21.695266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55470 (local address 127.0.97.60:36811)
0620 02:55:21.695453 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:21.695460 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:21.695523 (+ 63us) server_negotiation.cc:407] Connection header received
0620 02:55:21.695727 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:21.695732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:21.695841 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:21.695953 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:21.696819 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.697584 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.698866 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:21.699314 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:21.699452 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:21.702617 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:21.702621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:21.702624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:21.702654 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:21.706600 (+ 3946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:21.707544 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:21.707549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:21.707552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:21.707727 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:21.708205 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:21.708210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:21.708213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:21.708723 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:21.710482 (+ 1759us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:21.710922 (+ 440us) server_negotiation.cc:299] Negotiation successful
0620 02:55:21.711182 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 02:55:21.713429 14090 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dad7eeb782c49e09fcf2528c14c6162" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38529 } } peers { permanent_uuid: "ed830c43b38a4894b5b920c995c73c0c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38517 } } peers { permanent_uuid: "dfa74d2d3bb843e88392fc5aa1888221" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36811 } }
I20260620 02:55:21.717195 14037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5dad7eeb782c49e09fcf2528c14c6162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa74d2d3bb843e88392fc5aa1888221"
I20260620 02:55:21.717413 14037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfa74d2d3bb843e88392fc5aa1888221 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:55:21.719739 13987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed830c43b38a4894b5b920c995c73c0c: Bootstrap starting.
I20260620 02:55:21.720669 14037 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dfa74d2d3bb843e88392fc5aa1888221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dad7eeb782c49e09fcf2528c14c6162 in term 1.
W20260620 02:55:21.721149 14016 tablet.cc:2369] T 00000000000000000000000000000000 P dfa74d2d3bb843e88392fc5aa1888221: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:55:21.722033 13863 leader_election.cc:302] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dad7eeb782c49e09fcf2528c14c6162, dfa74d2d3bb843e88392fc5aa1888221; no voters:
I20260620 02:55:21.723628 14090 leader_election.cc:288] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162 [CANDIDATE]: Term 1 election: Requested vote from peers ed830c43b38a4894b5b920c995c73c0c (127.0.97.61:38517), dfa74d2d3bb843e88392fc5aa1888221 (127.0.97.60:36811)
I20260620 02:55:21.723858 14090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:55:21.724030 14090 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162 [term 1 LEADER]: Becoming Leader. State: Replica: 5dad7eeb782c49e09fcf2528c14c6162, State: Running, Role: LEADER
I20260620 02:55:21.724594 14090 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dad7eeb782c49e09fcf2528c14c6162" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38529 } } peers { permanent_uuid: "ed830c43b38a4894b5b920c995c73c0c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38517 } } peers { permanent_uuid: "dfa74d2d3bb843e88392fc5aa1888221" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36811 } }
I20260620 02:55:21.725571 13987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed830c43b38a4894b5b920c995c73c0c: Neither blocks nor log segments found. Creating new log.
I20260620 02:55:21.728215 13987 log.cc:824] T 00000000000000000000000000000000 P ed830c43b38a4894b5b920c995c73c0c: Log is configured to *not* fsync() on all Append() calls
I20260620 02:55:21.741788 13987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed830c43b38a4894b5b920c995c73c0c: No bootstrap required, opened a new log
I20260620 02:55:21.746522 14103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dad7eeb782c49e09fcf2528c14c6162. Latest consensus state: current_term: 1 leader_uuid: "5dad7eeb782c49e09fcf2528c14c6162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dad7eeb782c49e09fcf2528c14c6162" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38529 } } peers { permanent_uuid: "ed830c43b38a4894b5b920c995c73c0c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38517 } } peers { permanent_uuid: "dfa74d2d3bb843e88392fc5aa1888221" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36811 } } }
I20260620 02:55:21.747021 14103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162 [sys.catalog]: This master's current role is: LEADER
I20260620 02:55:21.751384 14106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:55:21.753793 14106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:55:21.761243 13987 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed830c43b38a4894b5b920c995c73c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dad7eeb782c49e09fcf2528c14c6162" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38529 } } peers { permanent_uuid: "ed830c43b38a4894b5b920c995c73c0c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38517 } } peers { permanent_uuid: "dfa74d2d3bb843e88392fc5aa1888221" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36811 } }
I20260620 02:55:21.762035 13987 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed830c43b38a4894b5b920c995c73c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:55:21.762308 13987 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed830c43b38a4894b5b920c995c73c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed830c43b38a4894b5b920c995c73c0c, State: Initialized, Role: FOLLOWER
I20260620 02:55:21.763516 13987 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed830c43b38a4894b5b920c995c73c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dad7eeb782c49e09fcf2528c14c6162" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38529 } } peers { permanent_uuid: "ed830c43b38a4894b5b920c995c73c0c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38517 } } peers { permanent_uuid: "dfa74d2d3bb843e88392fc5aa1888221" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36811 } }
I20260620 02:55:21.768465 14037 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dfa74d2d3bb843e88392fc5aa1888221 [term 1 FOLLOWER]: Refusing update from remote peer 5dad7eeb782c49e09fcf2528c14c6162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:55:21.769613 14103 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa74d2d3bb843e88392fc5aa1888221" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:55:21.774961 13962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5dad7eeb782c49e09fcf2528c14c6162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed830c43b38a4894b5b920c995c73c0c" is_pre_election: true
I20260620 02:55:21.775632 13962 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ed830c43b38a4894b5b920c995c73c0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dad7eeb782c49e09fcf2528c14c6162 in term 0.
I20260620 02:55:21.776113 13961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5dad7eeb782c49e09fcf2528c14c6162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed830c43b38a4894b5b920c995c73c0c"
I20260620 02:55:21.776360 13961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed830c43b38a4894b5b920c995c73c0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:55:21.780465 13961 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ed830c43b38a4894b5b920c995c73c0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dad7eeb782c49e09fcf2528c14c6162 in term 1.
I20260620 02:55:21.781183 13962 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ed830c43b38a4894b5b920c995c73c0c [term 1 FOLLOWER]: Refusing update from remote peer 5dad7eeb782c49e09fcf2528c14c6162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:55:21.782680 13987 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed830c43b38a4894b5b920c995c73c0c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:55:21.784256 14108 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed830c43b38a4894b5b920c995c73c0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dad7eeb782c49e09fcf2528c14c6162" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38529 } } peers { permanent_uuid: "ed830c43b38a4894b5b920c995c73c0c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38517 } } peers { permanent_uuid: "dfa74d2d3bb843e88392fc5aa1888221" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36811 } } }
I20260620 02:55:21.784698 14108 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed830c43b38a4894b5b920c995c73c0c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:55:21.785219 14103 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed830c43b38a4894b5b920c995c73c0c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:55:21.804293 14075 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfa74d2d3bb843e88392fc5aa1888221 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dad7eeb782c49e09fcf2528c14c6162. Latest consensus state: current_term: 1 leader_uuid: "5dad7eeb782c49e09fcf2528c14c6162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dad7eeb782c49e09fcf2528c14c6162" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38529 } } peers { permanent_uuid: "ed830c43b38a4894b5b920c995c73c0c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38517 } } peers { permanent_uuid: "dfa74d2d3bb843e88392fc5aa1888221" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36811 } } }
I20260620 02:55:21.804730 14075 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfa74d2d3bb843e88392fc5aa1888221 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:55:21.807575 14108 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed830c43b38a4894b5b920c995c73c0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dad7eeb782c49e09fcf2528c14c6162. Latest consensus state: current_term: 1 leader_uuid: "5dad7eeb782c49e09fcf2528c14c6162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dad7eeb782c49e09fcf2528c14c6162" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38529 } } peers { permanent_uuid: "ed830c43b38a4894b5b920c995c73c0c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38517 } } peers { permanent_uuid: "dfa74d2d3bb843e88392fc5aa1888221" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36811 } } }
I20260620 02:55:21.808003 14108 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed830c43b38a4894b5b920c995c73c0c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:55:21.808830 14107 mvcc.cc:205] Tried to move back new op lower bound from 7298761202730483712 to 7298761202598600704. Current Snapshot: MvccSnapshot[applied={T|T < 7298761202730483712}]
I20260620 02:55:21.812363 14109 mvcc.cc:205] Tried to move back new op lower bound from 7298761202730483712 to 7298761202598600704. Current Snapshot: MvccSnapshot[applied={T|T < 7298761202730483712}]
I20260620 02:55:21.815506 14112 mvcc.cc:205] Tried to move back new op lower bound from 7298761202730483712 to 7298761202598600704. Current Snapshot: MvccSnapshot[applied={T|T < 7298761202730483712}]
I20260620 02:55:21.815623 14108 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed830c43b38a4894b5b920c995c73c0c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5dad7eeb782c49e09fcf2528c14c6162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dad7eeb782c49e09fcf2528c14c6162" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38529 } } peers { permanent_uuid: "ed830c43b38a4894b5b920c995c73c0c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38517 } } peers { permanent_uuid: "dfa74d2d3bb843e88392fc5aa1888221" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36811 } } }
I20260620 02:55:21.815866 14108 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed830c43b38a4894b5b920c995c73c0c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:55:21.822448 14090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5dad7eeb782c49e09fcf2528c14c6162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dad7eeb782c49e09fcf2528c14c6162" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38529 } } peers { permanent_uuid: "ed830c43b38a4894b5b920c995c73c0c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38517 } } peers { permanent_uuid: "dfa74d2d3bb843e88392fc5aa1888221" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36811 } } }
I20260620 02:55:21.822901 14090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162 [sys.catalog]: This master's current role is: LEADER
I20260620 02:55:21.823220 14090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5dad7eeb782c49e09fcf2528c14c6162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dad7eeb782c49e09fcf2528c14c6162" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38529 } } peers { permanent_uuid: "ed830c43b38a4894b5b920c995c73c0c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38517 } } peers { permanent_uuid: "dfa74d2d3bb843e88392fc5aa1888221" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36811 } } }
I20260620 02:55:21.823522 14090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162 [sys.catalog]: This master's current role is: LEADER
I20260620 02:55:21.824541 14106 catalog_manager.cc:1347] Generated new cluster ID: 0f1f903e3a694891b9128ad07402524d
I20260620 02:55:21.824669 14106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:55:21.827729 14075 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfa74d2d3bb843e88392fc5aa1888221 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5dad7eeb782c49e09fcf2528c14c6162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dad7eeb782c49e09fcf2528c14c6162" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38529 } } peers { permanent_uuid: "ed830c43b38a4894b5b920c995c73c0c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38517 } } peers { permanent_uuid: "dfa74d2d3bb843e88392fc5aa1888221" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36811 } } }
I20260620 02:55:21.827987 14075 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfa74d2d3bb843e88392fc5aa1888221 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:55:21.873008 14106 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:55:21.876161 14106 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:55:21.894652 14106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5dad7eeb782c49e09fcf2528c14c6162: Generated new TSK 0
I20260620 02:55:21.895812 14106 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:55:21.965234 14126 catalog_manager.cc:1259] Loaded cluster ID: 0f1f903e3a694891b9128ad07402524d
I20260620 02:55:21.965410 14126 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ed830c43b38a4894b5b920c995c73c0c: loading cluster ID for follower catalog manager: success
I20260620 02:55:21.969861 14126 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ed830c43b38a4894b5b920c995c73c0c: acquiring CA information for follower catalog manager: success
I20260620 02:55:21.973527 14126 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ed830c43b38a4894b5b920c995c73c0c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:55:22.620187 14087 catalog_manager.cc:1259] Loaded cluster ID: 0f1f903e3a694891b9128ad07402524d
I20260620 02:55:22.620298 14087 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dfa74d2d3bb843e88392fc5aa1888221: loading cluster ID for follower catalog manager: success
I20260620 02:55:22.622822 14087 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dfa74d2d3bb843e88392fc5aa1888221: acquiring CA information for follower catalog manager: success
I20260620 02:55:22.624604 14087 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dfa74d2d3bb843e88392fc5aa1888221: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:55:23.045463 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 13846
I20260620 02:55:23.047247 13908 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:55:23.165769 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 13846
W20260620 02:55:23.186342 14014 connection.cc:535] server connection from 127.0.0.1:55464 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 02:55:23.186704 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 13920
W20260620 02:55:23.186311 13937 connection.cc:535] server connection from 127.0.0.1:55336 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 02:55:23.187794 13982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:55:23.295712 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 13920
I20260620 02:55:23.316754 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 13995
I20260620 02:55:23.317824 14057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:55:23.427932 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 13995
2026-06-20T02:55:23Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4810 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:55:23 dist-test-slave-8p8c krb5kdc[14141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:55:23 dist-test-slave-8p8c krb5kdc[14141](info): set up 2 sockets
Jun 20 02:55:23 dist-test-slave-8p8c krb5kdc[14141](info): commencing operation
krb5kdc: starting...
W20260620 02:55:24.574904 388 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:55:24 dist-test-slave-8p8c krb5kdc[14141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:55:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:55:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:55:24.814847 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43829
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:55:25.149040 14154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:55:25.149328 14154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:55:25.149420 14154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:55:25.156533 14154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:55:25.156630 14154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:55:25.156677 14154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:55:25.156728 14154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:55:25.156772 14154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:55:25.166383 14154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:55:25.167685 14154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:55:25.168979 14154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:55:25.180881 14159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:25.180948 14160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:25.180948 14162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:55:25.181907 14154 server_base.cc:1034] running on GCE node
I20260620 02:55:25.182789 14154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:55:25.184273 14154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:55:25.185540 14154 hybrid_clock.cc:648] HybridClock initialized: now 1781924125185459 us; error 51 us; skew 500 ppm
I20260620 02:55:25.185866 14154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:55:25 dist-test-slave-8p8c krb5kdc[14141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:55:25.189275 14154 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:55:25.190913 14154 webserver.cc:460] Webserver started at http://127.0.97.62:33959/ using document root <none> and password file <none>
I20260620 02:55:25.191633 14154 fs_manager.cc:359] Metadata directory not provided
I20260620 02:55:25.191759 14154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:55:25.192108 14154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:55:25.195207 14154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "061f68a21ed64aba809c4efc6c434944"
format_stamp: "Formatted at 2026-06-20 02:55:25 on dist-test-slave-8p8c"
server_key: "296a4f95958d36201e1164b231d9cebe"
server_key_iv: "43317c9266e36c6ba5f6c1810e129ed4"
server_key_version: "encryptionkey@0"
I20260620 02:55:25.196012 14154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "061f68a21ed64aba809c4efc6c434944"
format_stamp: "Formatted at 2026-06-20 02:55:25 on dist-test-slave-8p8c"
server_key: "296a4f95958d36201e1164b231d9cebe"
server_key_iv: "43317c9266e36c6ba5f6c1810e129ed4"
server_key_version: "encryptionkey@0"
I20260620 02:55:25.201423 14154 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:55:25.205214 14169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:25.206754 14154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:55:25.206950 14154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "061f68a21ed64aba809c4efc6c434944"
format_stamp: "Formatted at 2026-06-20 02:55:25 on dist-test-slave-8p8c"
server_key: "296a4f95958d36201e1164b231d9cebe"
server_key_iv: "43317c9266e36c6ba5f6c1810e129ed4"
server_key_version: "encryptionkey@0"
I20260620 02:55:25.207167 14154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:55:25.222836 14154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:55:25.227618 14154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:55:25.227931 14154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:55:25.247772 14154 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43829
I20260620 02:55:25.247802 14220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43829 every 8 connection(s)
I20260620 02:55:25.249224 14154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:55:25.253110 14221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:55:25.258140 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 14154
I20260620 02:55:25.258384 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:55:25.259197 388 external_mini_cluster.cc:1428] Setting key 034065bfbfa71c0a343b4e981bf3e494
I20260620 02:55:25.264976 14221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 061f68a21ed64aba809c4efc6c434944: Bootstrap starting.
I20260620 02:55:25.268220 14221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 061f68a21ed64aba809c4efc6c434944: Neither blocks nor log segments found. Creating new log.
I20260620 02:55:25.269235 14221 log.cc:824] T 00000000000000000000000000000000 P 061f68a21ed64aba809c4efc6c434944: Log is configured to *not* fsync() on all Append() calls
I20260620 02:55:25.272825 14221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 061f68a21ed64aba809c4efc6c434944: No bootstrap required, opened a new log
Jun 20 02:55:25 dist-test-slave-8p8c krb5kdc[14141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:55:25.283612 14221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 061f68a21ed64aba809c4efc6c434944 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "061f68a21ed64aba809c4efc6c434944" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43829 } }
I20260620 02:55:25.284099 14221 raft_consensus.cc:374] T 00000000000000000000000000000000 P 061f68a21ed64aba809c4efc6c434944 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:55:25.284210 14221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 061f68a21ed64aba809c4efc6c434944 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 061f68a21ed64aba809c4efc6c434944, State: Initialized, Role: FOLLOWER
I20260620 02:55:25.284876 14224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:25.261193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54314 (local address 127.0.97.62:43829)
0620 02:55:25.262106 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:25.262117 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:25.262159 (+ 42us) server_negotiation.cc:407] Connection header received
0620 02:55:25.264023 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:25.264046 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:25.264453 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:25.264859 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:25.266340 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:25.267715 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:25.269153 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:25.270156 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:25.270420 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:25.275448 (+ 5028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:25.275461 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:25.275488 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:25.275538 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:25.279817 (+ 4279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:25.280655 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:25.280665 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:25.280674 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:25.280901 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:25.281322 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:25.281327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:25.281331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:25.282148 (+ 817us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:25.282480 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:25.282949 (+ 469us) server_negotiation.cc:299] Negotiation successful
0620 02:55:25.283433 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":418,"threads_started":1}
I20260620 02:55:25.284915 14221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 061f68a21ed64aba809c4efc6c434944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "061f68a21ed64aba809c4efc6c434944" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43829 } }
I20260620 02:55:25.285218 14221 raft_consensus.cc:388] T 00000000000000000000000000000000 P 061f68a21ed64aba809c4efc6c434944 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:55:25.285363 14221 raft_consensus.cc:482] T 00000000000000000000000000000000 P 061f68a21ed64aba809c4efc6c434944 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:55:25.285689 14221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 061f68a21ed64aba809c4efc6c434944 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:55:25.288704 14221 raft_consensus.cc:504] T 00000000000000000000000000000000 P 061f68a21ed64aba809c4efc6c434944 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "061f68a21ed64aba809c4efc6c434944" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43829 } }
I20260620 02:55:25.289546 14221 leader_election.cc:302] T 00000000000000000000000000000000 P 061f68a21ed64aba809c4efc6c434944 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 061f68a21ed64aba809c4efc6c434944; no voters:
I20260620 02:55:25.289788 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:55:25.290292 14221 leader_election.cc:288] T 00000000000000000000000000000000 P 061f68a21ed64aba809c4efc6c434944 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:55:25.290431 14226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 061f68a21ed64aba809c4efc6c434944 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:55:25.292599 14226 raft_consensus.cc:686] T 00000000000000000000000000000000 P 061f68a21ed64aba809c4efc6c434944 [term 1 LEADER]: Becoming Leader. State: Replica: 061f68a21ed64aba809c4efc6c434944, State: Running, Role: LEADER
I20260620 02:55:25.292968 14221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 061f68a21ed64aba809c4efc6c434944 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:55:25.293227 14226 consensus_queue.cc:237] T 00000000000000000000000000000000 P 061f68a21ed64aba809c4efc6c434944 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "061f68a21ed64aba809c4efc6c434944" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43829 } }
I20260620 02:55:25.296871 14227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 061f68a21ed64aba809c4efc6c434944 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "061f68a21ed64aba809c4efc6c434944" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "061f68a21ed64aba809c4efc6c434944" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43829 } } }
I20260620 02:55:25.296929 14228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 061f68a21ed64aba809c4efc6c434944 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 061f68a21ed64aba809c4efc6c434944. Latest consensus state: current_term: 1 leader_uuid: "061f68a21ed64aba809c4efc6c434944" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "061f68a21ed64aba809c4efc6c434944" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43829 } } }
I20260620 02:55:25.297297 14227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 061f68a21ed64aba809c4efc6c434944 [sys.catalog]: This master's current role is: LEADER
I20260620 02:55:25.297341 14228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 061f68a21ed64aba809c4efc6c434944 [sys.catalog]: This master's current role is: LEADER
I20260620 02:55:25.299393 14234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:55:25.304759 14234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:55:25.314221 14234 catalog_manager.cc:1347] Generated new cluster ID: e3da82a8c8224b99a4d5a09423f52263
I20260620 02:55:25.314399 14234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:55:25.324828 14234 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:55:25.326218 14234 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:55:25.335484 14234 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 061f68a21ed64aba809c4efc6c434944: Generated new TSK 0
I20260620 02:55:25.336442 14234 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:55:25.351898 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 14154
I20260620 02:55:25.353205 14216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:55:25.447180 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 14154
2026-06-20T02:55:25Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2014 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:55:25 dist-test-slave-8p8c krb5kdc[14249](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:55:25 dist-test-slave-8p8c krb5kdc[14249](info): set up 2 sockets
Jun 20 02:55:25 dist-test-slave-8p8c krb5kdc[14249](info): commencing operation
krb5kdc: starting...
W20260620 02:55:26.562968 388 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:55:26 dist-test-slave-8p8c krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:55:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:55:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:55:26.807092 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45553
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45553
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:55:27.162966 14262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:55:27.163345 14262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:55:27.163511 14262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:55:27.171082 14262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:55:27.171268 14262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:55:27.171378 14262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:55:27.171478 14262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:55:27.171566 14262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:55:27.181795 14262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:55:27.183259 14262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:55:27.184604 14262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:55:27.197522 14267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:27.197558 14268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:27.197499 14270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:55:27.198289 14262 server_base.cc:1034] running on GCE node
I20260620 02:55:27.199105 14262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:55:27.200567 14262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:55:27.201823 14262 hybrid_clock.cc:648] HybridClock initialized: now 1781924127201774 us; error 40 us; skew 500 ppm
I20260620 02:55:27.202239 14262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:55:27 dist-test-slave-8p8c krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:55:27.208912 14262 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:55:27.210917 14262 webserver.cc:460] Webserver started at http://127.0.97.62:37747/ using document root <none> and password file <none>
I20260620 02:55:27.211644 14262 fs_manager.cc:359] Metadata directory not provided
I20260620 02:55:27.211794 14262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:55:27.212101 14262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:55:27.215246 14262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "9bf053f892fc4f9689fbb7cbabc2fc73"
format_stamp: "Formatted at 2026-06-20 02:55:27 on dist-test-slave-8p8c"
server_key: "257a01dbdb3f862d1684bece3c79d91e"
server_key_iv: "595294467f5b1a5b124125b75cdcf239"
server_key_version: "encryptionkey@0"
I20260620 02:55:27.216034 14262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "9bf053f892fc4f9689fbb7cbabc2fc73"
format_stamp: "Formatted at 2026-06-20 02:55:27 on dist-test-slave-8p8c"
server_key: "257a01dbdb3f862d1684bece3c79d91e"
server_key_iv: "595294467f5b1a5b124125b75cdcf239"
server_key_version: "encryptionkey@0"
I20260620 02:55:27.221328 14262 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260620 02:55:27.225010 14277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:27.226672 14262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:55:27.226891 14262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "9bf053f892fc4f9689fbb7cbabc2fc73"
format_stamp: "Formatted at 2026-06-20 02:55:27 on dist-test-slave-8p8c"
server_key: "257a01dbdb3f862d1684bece3c79d91e"
server_key_iv: "595294467f5b1a5b124125b75cdcf239"
server_key_version: "encryptionkey@0"
I20260620 02:55:27.227114 14262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:55:27.241878 14262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:55:27.246344 14262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:55:27.246731 14262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:55:27.265901 14262 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45553
I20260620 02:55:27.265933 14328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45553 every 8 connection(s)
I20260620 02:55:27.267362 14262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:55:27.270800 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 14262
I20260620 02:55:27.271023 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:55:27.271687 14329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:55:27.271852 388 external_mini_cluster.cc:1428] Setting key 0f502bf1f115ac073cae94e41653f334
I20260620 02:55:27.286610 14329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73: Bootstrap starting.
Jun 20 02:55:27 dist-test-slave-8p8c krb5kdc[14249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:55:27.289911 14329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73: Neither blocks nor log segments found. Creating new log.
I20260620 02:55:27.291007 14329 log.cc:824] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73: Log is configured to *not* fsync() on all Append() calls
I20260620 02:55:27.293815 14329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73: No bootstrap required, opened a new log
I20260620 02:55:27.295356 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:27.273906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49474 (local address 127.0.97.62:45553)
0620 02:55:27.274784 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:27.274795 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:27.274843 (+ 48us) server_negotiation.cc:407] Connection header received
0620 02:55:27.276536 (+ 1693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:27.276559 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:27.276964 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:27.277300 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:27.279109 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:27.280463 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:27.282316 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:27.283102 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:27.283247 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:27.288121 (+ 4874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:27.288130 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:27.288144 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:27.288185 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:27.291351 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:27.292031 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:27.292037 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:27.292043 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:27.292200 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:27.292604 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:27.292610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:27.292613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:27.293127 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:27.293423 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:27.293899 (+ 476us) server_negotiation.cc:299] Negotiation successful
0620 02:55:27.294323 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":422,"threads_started":1}
I20260620 02:55:27.301718 14329 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf053f892fc4f9689fbb7cbabc2fc73" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45553 } }
I20260620 02:55:27.302163 14329 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:55:27.302296 14329 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bf053f892fc4f9689fbb7cbabc2fc73, State: Initialized, Role: FOLLOWER
I20260620 02:55:27.302884 14329 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf053f892fc4f9689fbb7cbabc2fc73" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45553 } }
I20260620 02:55:27.303126 14329 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:55:27.303267 14329 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:55:27.303435 14329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:55:27.305513 14329 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf053f892fc4f9689fbb7cbabc2fc73" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45553 } }
I20260620 02:55:27.306001 14329 leader_election.cc:302] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bf053f892fc4f9689fbb7cbabc2fc73; no voters:
I20260620 02:55:27.306677 14329 leader_election.cc:288] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:55:27.306804 14334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:55:27.307852 14334 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [term 1 LEADER]: Becoming Leader. State: Replica: 9bf053f892fc4f9689fbb7cbabc2fc73, State: Running, Role: LEADER
I20260620 02:55:27.308419 14334 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf053f892fc4f9689fbb7cbabc2fc73" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45553 } }
I20260620 02:55:27.308897 14329 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:55:27.311481 14336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bf053f892fc4f9689fbb7cbabc2fc73. Latest consensus state: current_term: 1 leader_uuid: "9bf053f892fc4f9689fbb7cbabc2fc73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf053f892fc4f9689fbb7cbabc2fc73" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45553 } } }
I20260620 02:55:27.311383 14335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bf053f892fc4f9689fbb7cbabc2fc73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf053f892fc4f9689fbb7cbabc2fc73" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45553 } } }
I20260620 02:55:27.311753 14335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [sys.catalog]: This master's current role is: LEADER
I20260620 02:55:27.311753 14336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [sys.catalog]: This master's current role is: LEADER
I20260620 02:55:27.314265 14340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:55:27.320554 14340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:55:27.329315 14340 catalog_manager.cc:1347] Generated new cluster ID: 2316868da5294711abf39c1bfe59ff46
I20260620 02:55:27.329427 14340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:55:27.343191 14340 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:55:27.345288 14340 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:55:27.357637 14340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73: Generated new TSK 0
I20260620 02:55:27.358777 14340 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:55:27.424394 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45553
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:55:27.781586 14355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:55:27.781873 14355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:55:27.782035 14355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:55:27.789314 14355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:55:27.789448 14355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:55:27.789631 14355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:55:27.799607 14355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:55:27.800907 14355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:55:27.802227 14355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:55:27.815912 14363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:27.815955 14360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:27.815972 14361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:55:27.816821 14355 server_base.cc:1034] running on GCE node
I20260620 02:55:27.817524 14355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:55:27.818647 14355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:55:27.819902 14355 hybrid_clock.cc:648] HybridClock initialized: now 1781924127819825 us; error 62 us; skew 500 ppm
I20260620 02:55:27.820302 14355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:55:27 dist-test-slave-8p8c krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:55:27.823891 14355 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:55:27.825448 14355 webserver.cc:460] Webserver started at http://127.0.97.1:35225/ using document root <none> and password file <none>
I20260620 02:55:27.826120 14355 fs_manager.cc:359] Metadata directory not provided
I20260620 02:55:27.826223 14355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:55:27.826555 14355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:55:27.829480 14355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/data/instance:
uuid: "bebde3ed8c2b454495b46420279facf6"
format_stamp: "Formatted at 2026-06-20 02:55:27 on dist-test-slave-8p8c"
server_key: "bc2d1b09ff5e735d0c5af6bc5c626116"
server_key_iv: "ebd93faa9adea0754200db325bee04db"
server_key_version: "encryptionkey@0"
I20260620 02:55:27.830336 14355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/wal/instance:
uuid: "bebde3ed8c2b454495b46420279facf6"
format_stamp: "Formatted at 2026-06-20 02:55:27 on dist-test-slave-8p8c"
server_key: "bc2d1b09ff5e735d0c5af6bc5c626116"
server_key_iv: "ebd93faa9adea0754200db325bee04db"
server_key_version: "encryptionkey@0"
I20260620 02:55:27.835843 14355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260620 02:55:27.839903 14370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:27.841324 14355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:55:27.841496 14355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/wal
uuid: "bebde3ed8c2b454495b46420279facf6"
format_stamp: "Formatted at 2026-06-20 02:55:27 on dist-test-slave-8p8c"
server_key: "bc2d1b09ff5e735d0c5af6bc5c626116"
server_key_iv: "ebd93faa9adea0754200db325bee04db"
server_key_version: "encryptionkey@0"
I20260620 02:55:27.841722 14355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:55:27.873493 14355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:55:27.877801 14355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:55:27.878115 14355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:55:27.879158 14355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:55:27.880548 14355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:55:27.880620 14355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:27.880755 14355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:55:27.880816 14355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:27.916859 14355 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43863
I20260620 02:55:27.916989 14482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43863 every 8 connection(s)
I20260620 02:55:27.918210 14355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:55:27.921891 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 14355
I20260620 02:55:27.922122 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:55:27.922906 388 external_mini_cluster.cc:1428] Setting key 96073123d57459772670dc9676484b3c
Jun 20 02:55:27 dist-test-slave-8p8c krb5kdc[14249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:55:27.939256 14485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:27.920960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45007 (local address 127.0.97.62:45553)
0620 02:55:27.921505 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:27.921513 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:27.922662 (+ 1149us) server_negotiation.cc:407] Connection header received
0620 02:55:27.924781 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:27.924786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:27.924869 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:27.924995 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:27.926550 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:27.927291 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:27.928920 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:27.929331 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:27.929466 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:27.933287 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:27.933292 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:27.933296 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:27.933327 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:27.936083 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:27.936847 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:27.936855 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:27.936864 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:27.936989 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:27.937425 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:27.937429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:27.937432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:27.937710 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:27.937887 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:27.938643 (+ 756us) server_negotiation.cc:299] Negotiation successful
0620 02:55:27.938875 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":301,"threads_started":1}
I20260620 02:55:27.940327 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:27.921615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.1:45007)
0620 02:55:27.922466 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:27.922519 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:27.924466 (+ 1947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:27.925257 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:27.925266 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:27.925781 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:55:27.926336 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:27.926352 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:27.927465 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:27.927478 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:27.928760 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:27.928772 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:27.929570 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:27.929588 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:27.929809 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:27.930638 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:55:27.930660 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:55:27.933043 (+ 2383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:55:27.936255 (+ 3212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:27.936264 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:27.936281 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:27.936711 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:27.937100 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:27.937106 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:27.937110 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:27.937303 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:27.937844 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:55:27.937852 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:55:27.938213 (+ 361us) client_negotiation.cc:769] Sending connection context
0620 02:55:27.938533 (+ 320us) client_negotiation.cc:240] Negotiation successful
0620 02:55:27.939108 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":387,"threads_started":1}
I20260620 02:55:27.942054 14483 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45553
I20260620 02:55:27.942443 14483 heartbeater.cc:464] Registering TS with master...
I20260620 02:55:27.943400 14483 heartbeater.cc:511] Master 127.0.97.62:45553 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:55:27.946296 14294 ts_manager.cc:194] Registered new tserver with Master: bebde3ed8c2b454495b46420279facf6 (127.0.97.1:43863)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 02:55:27.948318 14294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45007
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:55:27.979377 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45553
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:55:28.306182 14489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:55:28.306568 14489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:55:28.306759 14489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:55:28.314245 14489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:55:28.314356 14489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:55:28.314566 14489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:55:28.324757 14489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:55:28.326028 14489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:55:28.327440 14489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:55:28.341184 14494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:28.341635 14497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:28.341801 14495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:55:28.342252 14489 server_base.cc:1034] running on GCE node
I20260620 02:55:28.342921 14489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:55:28.343943 14489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:55:28.345178 14489 hybrid_clock.cc:648] HybridClock initialized: now 1781924128345139 us; error 38 us; skew 500 ppm
I20260620 02:55:28.345517 14489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:55:28 dist-test-slave-8p8c krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:55:28.348891 14489 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:55:28.350340 14489 webserver.cc:460] Webserver started at http://127.0.97.2:38075/ using document root <none> and password file <none>
I20260620 02:55:28.351089 14489 fs_manager.cc:359] Metadata directory not provided
I20260620 02:55:28.351234 14489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:55:28.351541 14489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:55:28.354496 14489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/data/instance:
uuid: "0460c393c90f4abd993a05c6fe147550"
format_stamp: "Formatted at 2026-06-20 02:55:28 on dist-test-slave-8p8c"
server_key: "f945092c352ae6a3a3ded08bc13a83b3"
server_key_iv: "677ebaa8a80fdb52c177e647a71d42dc"
server_key_version: "encryptionkey@0"
I20260620 02:55:28.355273 14489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0460c393c90f4abd993a05c6fe147550"
format_stamp: "Formatted at 2026-06-20 02:55:28 on dist-test-slave-8p8c"
server_key: "f945092c352ae6a3a3ded08bc13a83b3"
server_key_iv: "677ebaa8a80fdb52c177e647a71d42dc"
server_key_version: "encryptionkey@0"
I20260620 02:55:28.360301 14489 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 02:55:28.363756 14504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:28.365214 14489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:55:28.365391 14489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/wal
uuid: "0460c393c90f4abd993a05c6fe147550"
format_stamp: "Formatted at 2026-06-20 02:55:28 on dist-test-slave-8p8c"
server_key: "f945092c352ae6a3a3ded08bc13a83b3"
server_key_iv: "677ebaa8a80fdb52c177e647a71d42dc"
server_key_version: "encryptionkey@0"
I20260620 02:55:28.365590 14489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:55:28.381806 14489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:55:28.386241 14489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:55:28.386587 14489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:55:28.387607 14489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:55:28.389019 14489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:55:28.389125 14489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:28.389273 14489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:55:28.389344 14489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:28.425019 14489 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38251
I20260620 02:55:28.425057 14616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38251 every 8 connection(s)
I20260620 02:55:28.426782 14489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:55:28.432541 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 14489
I20260620 02:55:28.432726 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:55:28.433370 388 external_mini_cluster.cc:1428] Setting key d36f23061f00cc8989f4faa1eb10a999
Jun 20 02:55:28 dist-test-slave-8p8c krb5kdc[14249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:55:28.449805 14485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:28.430079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59811 (local address 127.0.97.62:45553)
0620 02:55:28.430304 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:28.430309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:28.431923 (+ 1614us) server_negotiation.cc:407] Connection header received
0620 02:55:28.434552 (+ 2629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:28.434559 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:28.434656 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:28.434771 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:28.436514 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:28.437733 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:28.439363 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:28.440226 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:28.440377 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:28.443925 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:28.443930 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:28.443933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:28.443966 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:28.446881 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:28.447581 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:28.447588 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:28.447596 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:28.447713 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:28.448137 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:28.448142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:28.448145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:28.448394 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:28.448548 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:28.449234 (+ 686us) server_negotiation.cc:299] Negotiation successful
0620 02:55:28.449459 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 02:55:28.450917 14619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:28.430677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.2:59811)
0620 02:55:28.431722 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:28.431781 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:28.434226 (+ 2445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:28.435066 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:28.435078 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:28.435766 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:55:28.436337 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:28.436354 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:28.437890 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:28.437894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:28.439212 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:28.439224 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:28.440444 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:28.440449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:28.440667 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:28.441427 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:55:28.441462 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:55:28.443693 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:55:28.447018 (+ 3325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:28.447026 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:28.447043 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:28.447451 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:28.447821 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:28.447827 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:28.447831 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:28.448016 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:28.448519 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:55:28.448526 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:55:28.448865 (+ 339us) client_negotiation.cc:769] Sending connection context
0620 02:55:28.449141 (+ 276us) client_negotiation.cc:240] Negotiation successful
0620 02:55:28.449662 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":382,"threads_started":1}
I20260620 02:55:28.452435 14617 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45553
I20260620 02:55:28.452747 14617 heartbeater.cc:464] Registering TS with master...
I20260620 02:55:28.453459 14617 heartbeater.cc:511] Master 127.0.97.62:45553 requested a full tablet report, sending...
I20260620 02:55:28.454900 14294 ts_manager.cc:194] Registered new tserver with Master: 0460c393c90f4abd993a05c6fe147550 (127.0.97.2:38251)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:55:28.455760 14294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:55:28.486266 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45553
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:55:28.820387 14622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:55:28.820688 14622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:55:28.820838 14622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:55:28.828480 14622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:55:28.828606 14622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:55:28.828805 14622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:55:28.839289 14622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:55:28.840682 14622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:55:28.842115 14622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:55:28.855845 14627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:28.855801 14630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:28.856168 14628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:55:28.857013 14622 server_base.cc:1034] running on GCE node
I20260620 02:55:28.857719 14622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:55:28.858999 14622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:55:28.860268 14622 hybrid_clock.cc:648] HybridClock initialized: now 1781924128860200 us; error 46 us; skew 500 ppm
I20260620 02:55:28.860673 14622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:55:28 dist-test-slave-8p8c krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:55:28.864269 14622 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:55:28.865805 14622 webserver.cc:460] Webserver started at http://127.0.97.3:36497/ using document root <none> and password file <none>
I20260620 02:55:28.866676 14622 fs_manager.cc:359] Metadata directory not provided
I20260620 02:55:28.866837 14622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:55:28.867175 14622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:55:28.870694 14622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/data/instance:
uuid: "6494b6ffa1464d9cb2f168f618805049"
format_stamp: "Formatted at 2026-06-20 02:55:28 on dist-test-slave-8p8c"
server_key: "bf7f702fe647445822cff42af22bcfde"
server_key_iv: "ac93eb6ff9762522ae5c9e7e5f1b6478"
server_key_version: "encryptionkey@0"
I20260620 02:55:28.871809 14622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6494b6ffa1464d9cb2f168f618805049"
format_stamp: "Formatted at 2026-06-20 02:55:28 on dist-test-slave-8p8c"
server_key: "bf7f702fe647445822cff42af22bcfde"
server_key_iv: "ac93eb6ff9762522ae5c9e7e5f1b6478"
server_key_version: "encryptionkey@0"
I20260620 02:55:28.878461 14622 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 02:55:28.882330 14637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:28.884060 14622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 02:55:28.884253 14622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/wal
uuid: "6494b6ffa1464d9cb2f168f618805049"
format_stamp: "Formatted at 2026-06-20 02:55:28 on dist-test-slave-8p8c"
server_key: "bf7f702fe647445822cff42af22bcfde"
server_key_iv: "ac93eb6ff9762522ae5c9e7e5f1b6478"
server_key_version: "encryptionkey@0"
I20260620 02:55:28.884505 14622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:55:28.899497 14622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:55:28.904134 14622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:55:28.904440 14622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:55:28.905470 14622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:55:28.906935 14622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:55:28.907042 14622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:28.907200 14622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:55:28.907279 14622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:28.943979 14622 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46245
I20260620 02:55:28.944041 14749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46245 every 8 connection(s)
I20260620 02:55:28.945932 14622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:55:28.950904 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 14622
I20260620 02:55:28.951103 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:55:28.951725 388 external_mini_cluster.cc:1428] Setting key 95555a05cc6d6e7208e5de00d801e5f4
I20260620 02:55:28.952798 14483 heartbeater.cc:503] Master 127.0.97.62:45553 was elected leader, sending a full tablet report...
Jun 20 02:55:28 dist-test-slave-8p8c krb5kdc[14249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:55:28.976094 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:28.954089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36499 (local address 127.0.97.62:45553)
0620 02:55:28.955487 (+ 1398us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:28.955492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:28.956413 (+ 921us) server_negotiation.cc:407] Connection header received
0620 02:55:28.958469 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:28.958475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:28.958558 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:28.958661 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:28.960246 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:28.961037 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:28.962937 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:28.963371 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:28.963474 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:28.967229 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:28.967234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:28.967237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:28.967271 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:28.971657 (+ 4386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:28.972611 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:28.972615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:28.972618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:28.972752 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:28.973884 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:28.973890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:28.973893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:28.974182 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:28.974311 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:28.975623 (+ 1312us) server_negotiation.cc:299] Negotiation successful
0620 02:55:28.975817 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1266,"thread_start_us":442,"threads_started":1}
I20260620 02:55:28.977378 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:28.955052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.3:36499)
0620 02:55:28.956216 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:28.956275 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:28.958129 (+ 1854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:28.958924 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:28.958933 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:28.959467 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:55:28.960071 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:28.960088 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:28.961380 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:28.961384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:28.962785 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:28.962798 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:28.963573 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:28.963578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:28.963785 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:28.964466 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:55:28.964492 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:55:28.966994 (+ 2502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:55:28.971879 (+ 4885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:28.971887 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:28.971902 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:28.972421 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:28.972969 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:28.972973 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:28.972976 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:28.973705 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:28.974436 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:55:28.974444 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:55:28.975016 (+ 572us) client_negotiation.cc:769] Sending connection context
0620 02:55:28.975408 (+ 392us) client_negotiation.cc:240] Negotiation successful
0620 02:55:28.976112 (+ 704us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"thread_start_us":643,"threads_started":1}
I20260620 02:55:28.979481 14750 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45553
I20260620 02:55:28.979843 14750 heartbeater.cc:464] Registering TS with master...
I20260620 02:55:28.980626 14750 heartbeater.cc:511] Master 127.0.97.62:45553 requested a full tablet report, sending...
I20260620 02:55:28.982596 14294 ts_manager.cc:194] Registered new tserver with Master: 6494b6ffa1464d9cb2f168f618805049 (127.0.97.3:46245)
I20260620 02:55:28.983644 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:55:28.983716 14294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36499
I20260620 02:55:29.003170 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:28.988787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49480 (local address 127.0.97.62:45553)
0620 02:55:28.989015 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:28.989022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:28.989489 (+ 467us) server_negotiation.cc:407] Connection header received
0620 02:55:28.989644 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:28.989647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:28.989724 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:28.989813 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:28.991414 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:28.992174 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:28.993748 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:28.994179 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:28.994294 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:28.997010 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:28.997016 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:28.997019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:28.997054 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:28.999929 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:29.000868 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:29.000872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:29.000875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:29.001029 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:29.001452 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:29.001456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:29.001458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:29.001815 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:29.002013 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:29.002717 (+ 704us) server_negotiation.cc:299] Negotiation successful
0620 02:55:29.002917 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 02:55:29.008586 14294 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:55:29.015923 14294 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:55:29.052204 14763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:29.028772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46245 (local address 127.0.0.1:48306)
0620 02:55:29.043197 (+ 14425us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:29.043218 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:29.043352 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:29.044151 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:29.044160 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:29.044196 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:55:29.044747 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:29.044766 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:29.047276 (+ 2510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:29.047281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:29.049607 (+ 2326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:29.049622 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:29.051189 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:29.051196 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:29.051390 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:29.051434 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 02:55:29.051522 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 02:55:29.051749 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14299,"thread_start_us":14125,"threads_started":1}
I20260620 02:55:29.052948 14766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:29.029326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48306 (local address 127.0.97.3:46245)
0620 02:55:29.042849 (+ 13523us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:29.042863 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:29.043483 (+ 620us) server_negotiation.cc:407] Connection header received
0620 02:55:29.043604 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:29.043611 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:29.043885 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:29.045037 (+ 1152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:55:29.045376 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:29.047118 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:29.049804 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:29.050966 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:29.052067 (+ 1101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:29.052344 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:29.052422 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 02:55:29.052590 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13277,"thread_start_us":13083,"threads_started":1}
I20260620 02:55:29.054208 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:29.028633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43863 (local address 127.0.0.1:58072)
0620 02:55:29.041525 (+ 12892us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:29.041627 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:29.041925 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:29.046266 (+ 4341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:29.046271 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:29.046296 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:55:29.046765 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:29.046777 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:29.048699 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:29.048704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:29.051079 (+ 2375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:29.051094 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:29.053372 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:29.053378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:29.053537 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:29.053569 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 02:55:29.053654 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 02:55:29.053780 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12465,"mutex_wait_us":40,"thread_start_us":14272,"threads_started":1}
I20260620 02:55:29.054291 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:29.028941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58072 (local address 127.0.97.1:43863)
0620 02:55:29.045365 (+ 16424us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:29.045377 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:29.045411 (+ 34us) server_negotiation.cc:407] Connection header received
0620 02:55:29.045514 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:29.045521 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:29.045803 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:29.046016 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:55:29.046956 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:29.048550 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:29.051620 (+ 3070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:29.053186 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:29.053325 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:29.053553 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:29.053737 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 02:55:29.053918 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16243,"thread_start_us":16106,"threads_started":1}
I20260620 02:55:29.054997 14767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:29.044108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49694 (local address 127.0.97.2:38251)
0620 02:55:29.045756 (+ 1648us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:29.045766 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:29.046488 (+ 722us) server_negotiation.cc:407] Connection header received
0620 02:55:29.046817 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:29.046825 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:29.047035 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:29.047231 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:55:29.048195 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:29.049656 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:29.052156 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:29.053594 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:29.053722 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:29.053895 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:29.054481 (+ 586us) server_negotiation.cc:299] Negotiation successful
0620 02:55:29.054666 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1413,"thread_start_us":1275,"threads_started":1}
I20260620 02:55:29.055855 14768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:29.045727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38251 (local address 127.0.0.1:49694)
0620 02:55:29.046312 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:29.046399 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:29.046584 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:29.047462 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:29.047465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:29.047494 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:55:29.048001 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:29.048013 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:29.049863 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:29.049869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:29.051974 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:29.051987 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:29.054045 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:29.054052 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:29.054232 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:29.054264 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 02:55:29.055242 (+ 978us) client_negotiation.cc:240] Negotiation successful
0620 02:55:29.055382 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":1,"thread_start_us":300,"threads_started":1}
I20260620 02:55:29.057107 14685 tablet_service.cc:1467] Processing CreateTablet for tablet d00f4af0b50947f284a642e69880d1df (DEFAULT_TABLE table=test-table [id=047ea0634f9448199a579fe949beff3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:55:29.058148 14418 tablet_service.cc:1467] Processing CreateTablet for tablet d00f4af0b50947f284a642e69880d1df (DEFAULT_TABLE table=test-table [id=047ea0634f9448199a579fe949beff3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:55:29.058749 14685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d00f4af0b50947f284a642e69880d1df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:55:29.059357 14418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d00f4af0b50947f284a642e69880d1df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:55:29.059474 14552 tablet_service.cc:1467] Processing CreateTablet for tablet d00f4af0b50947f284a642e69880d1df (DEFAULT_TABLE table=test-table [id=047ea0634f9448199a579fe949beff3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:55:29.060765 14552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d00f4af0b50947f284a642e69880d1df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:55:29.071504 14769 tablet_bootstrap.cc:492] T d00f4af0b50947f284a642e69880d1df P bebde3ed8c2b454495b46420279facf6: Bootstrap starting.
I20260620 02:55:29.073740 14770 tablet_bootstrap.cc:492] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049: Bootstrap starting.
I20260620 02:55:29.075945 14769 tablet_bootstrap.cc:654] T d00f4af0b50947f284a642e69880d1df P bebde3ed8c2b454495b46420279facf6: Neither blocks nor log segments found. Creating new log.
I20260620 02:55:29.076233 14771 tablet_bootstrap.cc:492] T d00f4af0b50947f284a642e69880d1df P 0460c393c90f4abd993a05c6fe147550: Bootstrap starting.
I20260620 02:55:29.077371 14770 tablet_bootstrap.cc:654] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049: Neither blocks nor log segments found. Creating new log.
I20260620 02:55:29.077816 14769 log.cc:824] T d00f4af0b50947f284a642e69880d1df P bebde3ed8c2b454495b46420279facf6: Log is configured to *not* fsync() on all Append() calls
I20260620 02:55:29.078689 14770 log.cc:824] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049: Log is configured to *not* fsync() on all Append() calls
I20260620 02:55:29.079423 14771 tablet_bootstrap.cc:654] T d00f4af0b50947f284a642e69880d1df P 0460c393c90f4abd993a05c6fe147550: Neither blocks nor log segments found. Creating new log.
I20260620 02:55:29.080319 14771 log.cc:824] T d00f4af0b50947f284a642e69880d1df P 0460c393c90f4abd993a05c6fe147550: Log is configured to *not* fsync() on all Append() calls
I20260620 02:55:29.081375 14769 tablet_bootstrap.cc:492] T d00f4af0b50947f284a642e69880d1df P bebde3ed8c2b454495b46420279facf6: No bootstrap required, opened a new log
I20260620 02:55:29.081657 14769 ts_tablet_manager.cc:1397] T d00f4af0b50947f284a642e69880d1df P bebde3ed8c2b454495b46420279facf6: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260620 02:55:29.082991 14770 tablet_bootstrap.cc:492] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049: No bootstrap required, opened a new log
I20260620 02:55:29.083237 14771 tablet_bootstrap.cc:492] T d00f4af0b50947f284a642e69880d1df P 0460c393c90f4abd993a05c6fe147550: No bootstrap required, opened a new log
I20260620 02:55:29.083254 14770 ts_tablet_manager.cc:1397] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260620 02:55:29.083527 14771 ts_tablet_manager.cc:1397] T d00f4af0b50947f284a642e69880d1df P 0460c393c90f4abd993a05c6fe147550: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260620 02:55:29.090034 14769 raft_consensus.cc:348] T d00f4af0b50947f284a642e69880d1df P bebde3ed8c2b454495b46420279facf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebde3ed8c2b454495b46420279facf6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } } peers { permanent_uuid: "6494b6ffa1464d9cb2f168f618805049" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46245 } } peers { permanent_uuid: "0460c393c90f4abd993a05c6fe147550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38251 } }
I20260620 02:55:29.090443 14769 raft_consensus.cc:374] T d00f4af0b50947f284a642e69880d1df P bebde3ed8c2b454495b46420279facf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:55:29.090582 14769 raft_consensus.cc:729] T d00f4af0b50947f284a642e69880d1df P bebde3ed8c2b454495b46420279facf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bebde3ed8c2b454495b46420279facf6, State: Initialized, Role: FOLLOWER
I20260620 02:55:29.091241 14769 consensus_queue.cc:260] T d00f4af0b50947f284a642e69880d1df P bebde3ed8c2b454495b46420279facf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebde3ed8c2b454495b46420279facf6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } } peers { permanent_uuid: "6494b6ffa1464d9cb2f168f618805049" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46245 } } peers { permanent_uuid: "0460c393c90f4abd993a05c6fe147550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38251 } }
I20260620 02:55:29.093076 14769 ts_tablet_manager.cc:1428] T d00f4af0b50947f284a642e69880d1df P bebde3ed8c2b454495b46420279facf6: Time spent starting tablet: real 0.011s user 0.010s sys 0.001s
I20260620 02:55:29.094231 14771 raft_consensus.cc:348] T d00f4af0b50947f284a642e69880d1df P 0460c393c90f4abd993a05c6fe147550 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebde3ed8c2b454495b46420279facf6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } } peers { permanent_uuid: "6494b6ffa1464d9cb2f168f618805049" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46245 } } peers { permanent_uuid: "0460c393c90f4abd993a05c6fe147550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38251 } }
I20260620 02:55:29.094231 14770 raft_consensus.cc:348] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebde3ed8c2b454495b46420279facf6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } } peers { permanent_uuid: "6494b6ffa1464d9cb2f168f618805049" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46245 } } peers { permanent_uuid: "0460c393c90f4abd993a05c6fe147550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38251 } }
I20260620 02:55:29.094681 14770 raft_consensus.cc:374] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:55:29.094718 14771 raft_consensus.cc:374] T d00f4af0b50947f284a642e69880d1df P 0460c393c90f4abd993a05c6fe147550 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:55:29.094806 14770 raft_consensus.cc:729] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6494b6ffa1464d9cb2f168f618805049, State: Initialized, Role: FOLLOWER
I20260620 02:55:29.094831 14771 raft_consensus.cc:729] T d00f4af0b50947f284a642e69880d1df P 0460c393c90f4abd993a05c6fe147550 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0460c393c90f4abd993a05c6fe147550, State: Initialized, Role: FOLLOWER
I20260620 02:55:29.095412 14771 consensus_queue.cc:260] T d00f4af0b50947f284a642e69880d1df P 0460c393c90f4abd993a05c6fe147550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebde3ed8c2b454495b46420279facf6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } } peers { permanent_uuid: "6494b6ffa1464d9cb2f168f618805049" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46245 } } peers { permanent_uuid: "0460c393c90f4abd993a05c6fe147550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38251 } }
I20260620 02:55:29.095456 14770 consensus_queue.cc:260] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebde3ed8c2b454495b46420279facf6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } } peers { permanent_uuid: "6494b6ffa1464d9cb2f168f618805049" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46245 } } peers { permanent_uuid: "0460c393c90f4abd993a05c6fe147550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38251 } }
I20260620 02:55:29.097148 14617 heartbeater.cc:503] Master 127.0.97.62:45553 was elected leader, sending a full tablet report...
I20260620 02:55:29.097236 14770 ts_tablet_manager.cc:1428] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260620 02:55:29.097679 14771 ts_tablet_manager.cc:1428] T d00f4af0b50947f284a642e69880d1df P 0460c393c90f4abd993a05c6fe147550: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260620 02:55:29.098510 14750 heartbeater.cc:503] Master 127.0.97.62:45553 was elected leader, sending a full tablet report...
W20260620 02:55:29.170768 14484 tablet.cc:2369] T d00f4af0b50947f284a642e69880d1df P bebde3ed8c2b454495b46420279facf6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:55:29.179296 14618 tablet.cc:2369] T d00f4af0b50947f284a642e69880d1df P 0460c393c90f4abd993a05c6fe147550: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:55:29.204185 14751 tablet.cc:2369] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:55:29.216058 14777 raft_consensus.cc:482] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:55:29.216342 14777 raft_consensus.cc:504] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebde3ed8c2b454495b46420279facf6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } } peers { permanent_uuid: "6494b6ffa1464d9cb2f168f618805049" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46245 } } peers { permanent_uuid: "0460c393c90f4abd993a05c6fe147550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38251 } }
I20260620 02:55:29.218009 14777 leader_election.cc:288] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bebde3ed8c2b454495b46420279facf6 (127.0.97.1:43863), 0460c393c90f4abd993a05c6fe147550 (127.0.97.2:38251)
I20260620 02:55:29.224732 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:29.218502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55877 (local address 127.0.97.1:43863)
0620 02:55:29.218712 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:29.218718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:29.218740 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:55:29.218953 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:29.218958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:29.219034 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:29.219127 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:55:29.219747 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:29.220771 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:29.222672 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:29.223840 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:29.223950 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:29.224029 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:29.224404 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 02:55:29.224534 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 02:55:29.224790 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:29.218308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43863 (local address 127.0.97.3:55877)
0620 02:55:29.218565 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:29.218588 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:29.218761 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:29.219321 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:29.219327 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:29.219360 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:55:29.219547 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:29.219557 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:29.220908 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:29.220914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:29.222521 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:29.222533 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:29.224001 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:29.224007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:29.224206 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:29.224243 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 02:55:29.224348 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 02:55:29.224491 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 02:55:29.225896 14767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:29.219030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52227 (local address 127.0.97.2:38251)
0620 02:55:29.219260 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:29.219267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:29.219553 (+ 286us) server_negotiation.cc:407] Connection header received
0620 02:55:29.219705 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:29.219711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:29.219782 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:29.219850 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:55:29.220494 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:29.221453 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:29.223637 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:29.224816 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:29.224955 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:29.225078 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:29.225531 (+ 453us) server_negotiation.cc:299] Negotiation successful
0620 02:55:29.225716 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 02:55:29.225950 14778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:29.218780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38251 (local address 127.0.97.3:52227)
0620 02:55:29.219450 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:29.219465 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:29.219580 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:29.219894 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:29.219898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:29.219922 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:55:29.220308 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:29.220320 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:29.221633 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:29.221637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:29.223461 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:29.223476 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:29.225130 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:29.225136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:29.225309 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:29.225368 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 02:55:29.225487 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 02:55:29.225624 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":373,"threads_started":1}
I20260620 02:55:29.225967 14438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00f4af0b50947f284a642e69880d1df" candidate_uuid: "6494b6ffa1464d9cb2f168f618805049" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebde3ed8c2b454495b46420279facf6" is_pre_election: true
I20260620 02:55:29.226636 14438 raft_consensus.cc:2454] T d00f4af0b50947f284a642e69880d1df P bebde3ed8c2b454495b46420279facf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6494b6ffa1464d9cb2f168f618805049 in term 0.
I20260620 02:55:29.227159 14572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00f4af0b50947f284a642e69880d1df" candidate_uuid: "6494b6ffa1464d9cb2f168f618805049" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0460c393c90f4abd993a05c6fe147550" is_pre_election: true
I20260620 02:55:29.227486 14639 leader_election.cc:302] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6494b6ffa1464d9cb2f168f618805049, bebde3ed8c2b454495b46420279facf6; no voters:
I20260620 02:55:29.227748 14572 raft_consensus.cc:2454] T d00f4af0b50947f284a642e69880d1df P 0460c393c90f4abd993a05c6fe147550 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6494b6ffa1464d9cb2f168f618805049 in term 0.
I20260620 02:55:29.227984 14777 raft_consensus.cc:2781] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:55:29.228159 14777 raft_consensus.cc:482] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:55:29.228271 14777 raft_consensus.cc:3037] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:55:29.230657 14777 raft_consensus.cc:504] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebde3ed8c2b454495b46420279facf6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } } peers { permanent_uuid: "6494b6ffa1464d9cb2f168f618805049" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46245 } } peers { permanent_uuid: "0460c393c90f4abd993a05c6fe147550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38251 } }
I20260620 02:55:29.231233 14777 leader_election.cc:288] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049 [CANDIDATE]: Term 1 election: Requested vote from peers bebde3ed8c2b454495b46420279facf6 (127.0.97.1:43863), 0460c393c90f4abd993a05c6fe147550 (127.0.97.2:38251)
I20260620 02:55:29.231784 14438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00f4af0b50947f284a642e69880d1df" candidate_uuid: "6494b6ffa1464d9cb2f168f618805049" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebde3ed8c2b454495b46420279facf6"
I20260620 02:55:29.231933 14572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00f4af0b50947f284a642e69880d1df" candidate_uuid: "6494b6ffa1464d9cb2f168f618805049" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0460c393c90f4abd993a05c6fe147550"
I20260620 02:55:29.232019 14438 raft_consensus.cc:3037] T d00f4af0b50947f284a642e69880d1df P bebde3ed8c2b454495b46420279facf6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:55:29.232197 14572 raft_consensus.cc:3037] T d00f4af0b50947f284a642e69880d1df P 0460c393c90f4abd993a05c6fe147550 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:55:29.235381 14438 raft_consensus.cc:2454] T d00f4af0b50947f284a642e69880d1df P bebde3ed8c2b454495b46420279facf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6494b6ffa1464d9cb2f168f618805049 in term 1.
I20260620 02:55:29.235380 14572 raft_consensus.cc:2454] T d00f4af0b50947f284a642e69880d1df P 0460c393c90f4abd993a05c6fe147550 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6494b6ffa1464d9cb2f168f618805049 in term 1.
I20260620 02:55:29.235809 14639 leader_election.cc:302] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6494b6ffa1464d9cb2f168f618805049, bebde3ed8c2b454495b46420279facf6; no voters:
I20260620 02:55:29.236182 14777 raft_consensus.cc:2781] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:55:29.236693 14777 raft_consensus.cc:686] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049 [term 1 LEADER]: Becoming Leader. State: Replica: 6494b6ffa1464d9cb2f168f618805049, State: Running, Role: LEADER
I20260620 02:55:29.237260 14777 consensus_queue.cc:237] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebde3ed8c2b454495b46420279facf6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } } peers { permanent_uuid: "6494b6ffa1464d9cb2f168f618805049" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46245 } } peers { permanent_uuid: "0460c393c90f4abd993a05c6fe147550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38251 } }
I20260620 02:55:29.241999 14294 catalog_manager.cc:5495] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6494b6ffa1464d9cb2f168f618805049 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "6494b6ffa1464d9cb2f168f618805049" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bebde3ed8c2b454495b46420279facf6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6494b6ffa1464d9cb2f168f618805049" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0460c393c90f4abd993a05c6fe147550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38251 } health_report { overall_health: UNKNOWN } } }
I20260620 02:55:29.295221 14766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:29.289779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48320 (local address 127.0.97.3:46245)
0620 02:55:29.289986 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:29.289991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:29.290009 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:55:29.290109 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:29.290113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:29.290194 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:29.290331 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:55:29.290812 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:29.291684 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:29.293063 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:29.293508 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:29.293614 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:29.294152 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:55:29.294690 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:55:29.294808 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:29.294919 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 02:55:29.294986 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 02:55:29.303560 14572 raft_consensus.cc:1261] T d00f4af0b50947f284a642e69880d1df P 0460c393c90f4abd993a05c6fe147550 [term 1 FOLLOWER]: Refusing update from remote peer 6494b6ffa1464d9cb2f168f618805049: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:55:29.303594 14438 raft_consensus.cc:1261] T d00f4af0b50947f284a642e69880d1df P bebde3ed8c2b454495b46420279facf6 [term 1 FOLLOWER]: Refusing update from remote peer 6494b6ffa1464d9cb2f168f618805049: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:55:29.304540 14779 consensus_queue.cc:1035] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0460c393c90f4abd993a05c6fe147550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:55:29.304908 14777 consensus_queue.cc:1035] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bebde3ed8c2b454495b46420279facf6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:55:29.334784 14294 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:55:29.335084 14294 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:55:29.339012 14294 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73: Sending DeleteTablet for 3 replicas of tablet d00f4af0b50947f284a642e69880d1df
I20260620 02:55:29.340293 14685 tablet_service.cc:1514] Processing DeleteTablet for tablet d00f4af0b50947f284a642e69880d1df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:55:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48306
I20260620 02:55:29.340448 14418 tablet_service.cc:1514] Processing DeleteTablet for tablet d00f4af0b50947f284a642e69880d1df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:55:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58072
I20260620 02:55:29.340448 14552 tablet_service.cc:1514] Processing DeleteTablet for tablet d00f4af0b50947f284a642e69880d1df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:55:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49694
I20260620 02:55:29.341754 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 14355
I20260620 02:55:29.350512 14794 tablet_replica.cc:330] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049: stopping tablet replica
I20260620 02:55:29.351040 14794 raft_consensus.cc:2229] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:55:29.351863 14794 raft_consensus.cc:2258] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:55:29.353591 14793 tablet_replica.cc:330] T d00f4af0b50947f284a642e69880d1df P bebde3ed8c2b454495b46420279facf6: stopping tablet replica
I20260620 02:55:29.354104 14793 raft_consensus.cc:2229] T d00f4af0b50947f284a642e69880d1df P bebde3ed8c2b454495b46420279facf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:55:29.354846 14793 raft_consensus.cc:2258] T d00f4af0b50947f284a642e69880d1df P bebde3ed8c2b454495b46420279facf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:55:29.356149 14792 tablet_replica.cc:330] T d00f4af0b50947f284a642e69880d1df P 0460c393c90f4abd993a05c6fe147550: stopping tablet replica
I20260620 02:55:29.356655 14792 raft_consensus.cc:2229] T d00f4af0b50947f284a642e69880d1df P 0460c393c90f4abd993a05c6fe147550 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:55:29.357158 14793 ts_tablet_manager.cc:1905] T d00f4af0b50947f284a642e69880d1df P bebde3ed8c2b454495b46420279facf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:55:29.357198 14792 raft_consensus.cc:2258] T d00f4af0b50947f284a642e69880d1df P 0460c393c90f4abd993a05c6fe147550 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:55:29.358100 14794 ts_tablet_manager.cc:1905] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:55:29.359355 14792 ts_tablet_manager.cc:1905] T d00f4af0b50947f284a642e69880d1df P 0460c393c90f4abd993a05c6fe147550: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:55:29.364658 14793 ts_tablet_manager.cc:1918] T d00f4af0b50947f284a642e69880d1df P bebde3ed8c2b454495b46420279facf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:55:29.364867 14793 log.cc:1196] T d00f4af0b50947f284a642e69880d1df P bebde3ed8c2b454495b46420279facf6: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/wal/wals/d00f4af0b50947f284a642e69880d1df
I20260620 02:55:29.365434 14793 ts_tablet_manager.cc:1939] T d00f4af0b50947f284a642e69880d1df P bebde3ed8c2b454495b46420279facf6: Deleting consensus metadata
I20260620 02:55:29.366752 14792 ts_tablet_manager.cc:1918] T d00f4af0b50947f284a642e69880d1df P 0460c393c90f4abd993a05c6fe147550: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:55:29.367003 14792 log.cc:1196] T d00f4af0b50947f284a642e69880d1df P 0460c393c90f4abd993a05c6fe147550: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/wal/wals/d00f4af0b50947f284a642e69880d1df
I20260620 02:55:29.367657 14792 ts_tablet_manager.cc:1939] T d00f4af0b50947f284a642e69880d1df P 0460c393c90f4abd993a05c6fe147550: Deleting consensus metadata
I20260620 02:55:29.367617 14279 catalog_manager.cc:4841] TS bebde3ed8c2b454495b46420279facf6 (127.0.97.1:43863): tablet d00f4af0b50947f284a642e69880d1df (table test-table [id=047ea0634f9448199a579fe949beff3b]) successfully deleted
I20260620 02:55:29.369585 14279 catalog_manager.cc:4841] TS 0460c393c90f4abd993a05c6fe147550 (127.0.97.2:38251): tablet d00f4af0b50947f284a642e69880d1df (table test-table [id=047ea0634f9448199a579fe949beff3b]) successfully deleted
Jun 20 02:55:29 dist-test-slave-8p8c krb5kdc[14249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:55:29.372247 14794 ts_tablet_manager.cc:1918] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:55:29.372432 14794 log.cc:1196] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/wal/wals/d00f4af0b50947f284a642e69880d1df
I20260620 02:55:29.372988 14794 ts_tablet_manager.cc:1939] T d00f4af0b50947f284a642e69880d1df P 6494b6ffa1464d9cb2f168f618805049: Deleting consensus metadata
I20260620 02:55:29.374799 14280 catalog_manager.cc:4841] TS 6494b6ffa1464d9cb2f168f618805049 (127.0.97.3:46245): tablet d00f4af0b50947f284a642e69880d1df (table test-table [id=047ea0634f9448199a579fe949beff3b]) successfully deleted
I20260620 02:55:29.379251 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:29.343310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58084 (local address 127.0.97.1:43863)
0620 02:55:29.343557 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:29.343566 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:29.352798 (+ 9232us) server_negotiation.cc:407] Connection header received
0620 02:55:29.353187 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:29.353194 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:29.353387 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:29.353541 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:29.358354 (+ 4813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:29.359599 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:29.366497 (+ 6898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:29.367144 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:29.367250 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:29.372417 (+ 5167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:29.372434 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:29.372456 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:29.372526 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:29.375871 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:29.376816 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:29.376824 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:29.376827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:29.377007 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:29.377466 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:29.377470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:29.377472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:29.377839 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:29.378119 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:29.378595 (+ 476us) server_negotiation.cc:299] Negotiation successful
0620 02:55:29.378783 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 02:55:29.379961 14478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:55:29.491044 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 14355
I20260620 02:55:29.509454 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 14489
Jun 20 02:55:29 dist-test-slave-8p8c krb5kdc[14249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:55:29.527909 14767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:29.510267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49698 (local address 127.0.97.2:38251)
0620 02:55:29.510482 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:29.510489 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:29.510510 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:55:29.510691 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:29.510697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:29.510796 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:29.510909 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:29.512273 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:29.513417 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:29.515190 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:29.515827 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:29.515982 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:29.520955 (+ 4973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:29.520975 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:29.520997 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:29.521050 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:29.524865 (+ 3815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:29.525581 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:29.525593 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:29.525603 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:29.525757 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:29.526126 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:29.526131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:29.526135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:29.526731 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:29.527016 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:29.527309 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 02:55:29.527533 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:55:29.528661 14612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:55:29.621230 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 14489
I20260620 02:55:29.639462 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 14622
Jun 20 02:55:29 dist-test-slave-8p8c krb5kdc[14249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:55:29.658921 14766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:29.640351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48332 (local address 127.0.97.3:46245)
0620 02:55:29.640529 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:29.640534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:29.640633 (+ 99us) server_negotiation.cc:407] Connection header received
0620 02:55:29.640831 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:29.640835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:29.640903 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:29.640998 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:29.642857 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:29.643707 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:29.645639 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:29.646148 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:29.646251 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:29.652057 (+ 5806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:29.652072 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:29.652089 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:29.652136 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:29.655440 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:29.656363 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:29.656371 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:29.656373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:29.656565 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:29.657018 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:29.657022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:29.657024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:29.657407 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:29.657697 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:29.658138 (+ 441us) server_negotiation.cc:299] Negotiation successful
0620 02:55:29.658324 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 02:55:29.659855 14745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:55:29.783874 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 14622
I20260620 02:55:29.804512 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 14262
I20260620 02:55:29.805768 14324 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:55:29.919066 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 14262
I20260620 02:55:29.940984 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45553
--webserver_interface=127.0.97.62
--webserver_port=37747
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:55:30.275712 14800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:55:30.276015 14800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:55:30.276187 14800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:55:30.283519 14800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:55:30.283632 14800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:55:30.283746 14800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:55:30.283820 14800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:55:30.283886 14800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:55:30.294095 14800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=37747
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:55:30.295492 14800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:55:30.296866 14800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:55:30.321460 14807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:30.321442 14809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:30.321532 14806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:55:30.324277 14800 server_base.cc:1034] running on GCE node
I20260620 02:55:30.325263 14800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:55:30.327155 14800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:55:30.328461 14800 hybrid_clock.cc:648] HybridClock initialized: now 1781924130328385 us; error 40 us; skew 500 ppm
I20260620 02:55:30.328935 14800 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:55:30 dist-test-slave-8p8c krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:55:30.333214 14800 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:55:30.335266 14800 webserver.cc:460] Webserver started at http://127.0.97.62:37747/ using document root <none> and password file <none>
I20260620 02:55:30.336138 14800 fs_manager.cc:359] Metadata directory not provided
I20260620 02:55:30.336311 14800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:55:30.342957 14800 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 02:55:30.346310 14816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:30.348026 14800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:55:30.348307 14800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "9bf053f892fc4f9689fbb7cbabc2fc73"
format_stamp: "Formatted at 2026-06-20 02:55:27 on dist-test-slave-8p8c"
server_key: "257a01dbdb3f862d1684bece3c79d91e"
server_key_iv: "595294467f5b1a5b124125b75cdcf239"
server_key_version: "encryptionkey@0"
I20260620 02:55:30.349460 14800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:55:30.371001 14800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:55:30.375924 14800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:55:30.376281 14800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:55:30.397096 14800 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45553
I20260620 02:55:30.397114 14867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45553 every 8 connection(s)
I20260620 02:55:30.398783 14800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:55:30.405946 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 14800
I20260620 02:55:30.405982 14868 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:55:30.406981 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:43863
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45553
--webserver_port=35225
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:55:30.411249 14868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73: Bootstrap starting.
I20260620 02:55:30.435001 14868 log.cc:824] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73: Log is configured to *not* fsync() on all Append() calls
I20260620 02:55:30.447078 14868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:55:30.447629 14868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73: Bootstrap complete.
I20260620 02:55:30.457266 14868 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf053f892fc4f9689fbb7cbabc2fc73" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45553 } }
I20260620 02:55:30.458169 14868 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bf053f892fc4f9689fbb7cbabc2fc73, State: Initialized, Role: FOLLOWER
I20260620 02:55:30.458830 14868 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf053f892fc4f9689fbb7cbabc2fc73" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45553 } }
I20260620 02:55:30.459098 14868 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:55:30.459232 14868 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:55:30.459424 14868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:55:30.463382 14868 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf053f892fc4f9689fbb7cbabc2fc73" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45553 } }
I20260620 02:55:30.463907 14868 leader_election.cc:302] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bf053f892fc4f9689fbb7cbabc2fc73; no voters:
I20260620 02:55:30.464582 14868 leader_election.cc:288] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 02:55:30.464730 14872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 02:55:30.465867 14872 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [term 2 LEADER]: Becoming Leader. State: Replica: 9bf053f892fc4f9689fbb7cbabc2fc73, State: Running, Role: LEADER
I20260620 02:55:30.466518 14872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf053f892fc4f9689fbb7cbabc2fc73" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45553 } }
I20260620 02:55:30.467020 14868 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:55:30.469296 14873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9bf053f892fc4f9689fbb7cbabc2fc73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf053f892fc4f9689fbb7cbabc2fc73" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45553 } } }
I20260620 02:55:30.469627 14874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bf053f892fc4f9689fbb7cbabc2fc73. Latest consensus state: current_term: 2 leader_uuid: "9bf053f892fc4f9689fbb7cbabc2fc73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf053f892fc4f9689fbb7cbabc2fc73" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45553 } } }
I20260620 02:55:30.469877 14873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [sys.catalog]: This master's current role is: LEADER
I20260620 02:55:30.469969 14874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [sys.catalog]: This master's current role is: LEADER
I20260620 02:55:30.482146 14877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:55:30.491814 14877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:55:30.494019 14877 catalog_manager.cc:1259] Loaded cluster ID: 2316868da5294711abf39c1bfe59ff46
I20260620 02:55:30.494302 14877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:55:30.499683 14877 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 02:55:30.518327 14877 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 02:55:30.525913 14877 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:55:30.527360 14877 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73: Loaded TSK: 0
I20260620 02:55:30.528581 14877 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 02:55:30.800513 14870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:55:30.800822 14870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:55:30.800973 14870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:55:30.808441 14870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:55:30.808560 14870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:55:30.808728 14870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:55:30.818817 14870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:43863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=35225
--tserver_master_addrs=127.0.97.62:45553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:55:30.820200 14870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:55:30.821533 14870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:55:30.837270 14895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:30.837231 14898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:30.837318 14896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:55:30.838272 14870 server_base.cc:1034] running on GCE node
I20260620 02:55:30.838987 14870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:55:30.840137 14870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:55:30.841466 14870 hybrid_clock.cc:648] HybridClock initialized: now 1781924130841385 us; error 68 us; skew 500 ppm
I20260620 02:55:30.841845 14870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:55:30 dist-test-slave-8p8c krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:55:30.845593 14870 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:55:30.847376 14870 webserver.cc:460] Webserver started at http://127.0.97.1:35225/ using document root <none> and password file <none>
I20260620 02:55:30.848126 14870 fs_manager.cc:359] Metadata directory not provided
I20260620 02:55:30.848274 14870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:55:30.854153 14870 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260620 02:55:30.857146 14905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:30.858691 14870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:55:30.858860 14870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/wal
uuid: "bebde3ed8c2b454495b46420279facf6"
format_stamp: "Formatted at 2026-06-20 02:55:27 on dist-test-slave-8p8c"
server_key: "bc2d1b09ff5e735d0c5af6bc5c626116"
server_key_iv: "ebd93faa9adea0754200db325bee04db"
server_key_version: "encryptionkey@0"
I20260620 02:55:30.859833 14870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:55:30.876860 14870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:55:30.881836 14870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:55:30.882123 14870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:55:30.883256 14870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:55:30.884773 14870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:55:30.884893 14870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:30.885013 14870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:55:30.885116 14870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:30.923069 14870 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43863
I20260620 02:55:30.923092 15017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43863 every 8 connection(s)
I20260620 02:55:30.924541 14870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:55:30.934196 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 14870
I20260620 02:55:30.934881 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:38251
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45553
--webserver_port=38075
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:55:30 dist-test-slave-8p8c krb5kdc[14249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:55:30.950344 15021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:30.927824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.1:35059)
0620 02:55:30.928628 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:30.928676 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:30.930480 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:30.931987 (+ 1507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:30.931995 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:30.932582 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:55:30.933154 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:30.933171 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:30.935054 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:30.935067 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:30.936428 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:30.936441 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:30.937634 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:30.937642 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:30.937952 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:30.938932 (+ 980us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:55:30.938987 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:55:30.941509 (+ 2522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:55:30.945400 (+ 3891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:30.945409 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:30.945425 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:30.945900 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:30.946395 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:30.946403 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:30.946409 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:30.946730 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:30.947550 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:55:30.947558 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:55:30.947922 (+ 364us) client_negotiation.cc:769] Sending connection context
0620 02:55:30.948212 (+ 290us) client_negotiation.cc:240] Negotiation successful
0620 02:55:30.948810 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":351,"threads_started":1}
I20260620 02:55:30.950344 15020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:30.927629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35059 (local address 127.0.97.62:45553)
0620 02:55:30.928300 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:30.928313 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:30.928843 (+ 530us) server_negotiation.cc:407] Connection header received
0620 02:55:30.931038 (+ 2195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:30.931057 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:30.931422 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:30.931871 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:30.933404 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:30.934872 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:30.936600 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:30.937388 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:30.937606 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:30.941829 (+ 4223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:30.941838 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:30.941854 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:30.941898 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:30.945228 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:30.946055 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:30.946071 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:30.946077 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:30.946250 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:30.946876 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:30.946884 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:30.946888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:30.947383 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:30.947622 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:30.948445 (+ 823us) server_negotiation.cc:299] Negotiation successful
0620 02:55:30.948871 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":300,"threads_started":1}
I20260620 02:55:30.954658 15018 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45553
I20260620 02:55:30.955050 15018 heartbeater.cc:464] Registering TS with master...
I20260620 02:55:30.956243 15018 heartbeater.cc:511] Master 127.0.97.62:45553 requested a full tablet report, sending...
I20260620 02:55:30.959154 14833 ts_manager.cc:194] Registered new tserver with Master: bebde3ed8c2b454495b46420279facf6 (127.0.97.1:43863)
I20260620 02:55:30.961251 14833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35059
W20260620 02:55:31.282573 15022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:55:31.282905 15022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:55:31.283033 15022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:55:31.290608 15022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:55:31.290717 15022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:55:31.290831 15022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:55:31.300813 15022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:38251
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=38075
--tserver_master_addrs=127.0.97.62:45553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:55:31.302057 15022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:55:31.303349 15022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:55:31.316701 15027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:31.316841 15028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:31.317519 15030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:55:31.318512 15022 server_base.cc:1034] running on GCE node
I20260620 02:55:31.319134 15022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:55:31.320137 15022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:55:31.321339 15022 hybrid_clock.cc:648] HybridClock initialized: now 1781924131321295 us; error 35 us; skew 500 ppm
I20260620 02:55:31.321666 15022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:55:31 dist-test-slave-8p8c krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:55:31.324978 15022 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:55:31.326763 15022 webserver.cc:460] Webserver started at http://127.0.97.2:38075/ using document root <none> and password file <none>
I20260620 02:55:31.327458 15022 fs_manager.cc:359] Metadata directory not provided
I20260620 02:55:31.327570 15022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:55:31.333073 15022 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 02:55:31.336019 15037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:31.337634 15022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:55:31.337905 15022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/wal
uuid: "0460c393c90f4abd993a05c6fe147550"
format_stamp: "Formatted at 2026-06-20 02:55:28 on dist-test-slave-8p8c"
server_key: "f945092c352ae6a3a3ded08bc13a83b3"
server_key_iv: "677ebaa8a80fdb52c177e647a71d42dc"
server_key_version: "encryptionkey@0"
I20260620 02:55:31.339039 15022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:55:31.355330 15022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:55:31.359920 15022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:55:31.360219 15022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:55:31.361228 15022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:55:31.362592 15022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:55:31.362687 15022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:31.362841 15022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:55:31.362931 15022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:31.398433 15022 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38251
I20260620 02:55:31.398490 15149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38251 every 8 connection(s)
I20260620 02:55:31.399966 15022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:55:31.409857 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 15022
I20260620 02:55:31.410583 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:46245
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45553
--webserver_port=36497
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:55:31 dist-test-slave-8p8c krb5kdc[14249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:55:31.425328 15020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:31.402930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56243 (local address 127.0.97.62:45553)
0620 02:55:31.403197 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:31.403203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:31.404448 (+ 1245us) server_negotiation.cc:407] Connection header received
0620 02:55:31.407279 (+ 2831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:31.407283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:31.407396 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:31.407489 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:31.409412 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:31.410599 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:31.413428 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:31.414152 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:31.414284 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:31.419276 (+ 4992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:31.419290 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:31.419294 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:31.419329 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:31.422213 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:31.422994 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:31.423000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:31.423004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:31.423153 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:31.423602 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:31.423614 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:31.423619 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:31.423889 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:31.424047 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:31.424746 (+ 699us) server_negotiation.cc:299] Negotiation successful
0620 02:55:31.424981 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 02:55:31.426563 15152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:31.403480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.2:56243)
0620 02:55:31.404273 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:31.404319 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:31.406933 (+ 2614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:31.407757 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:31.407769 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:31.408458 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:55:31.409183 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:31.409205 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:31.410923 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:31.410931 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:31.413242 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:31.413261 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:31.414435 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:31.414446 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:31.414834 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:31.415976 (+ 1142us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:55:31.416011 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:55:31.419047 (+ 3036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:55:31.422410 (+ 3363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:31.422419 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:31.422436 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:31.422857 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:31.423276 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:31.423281 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:31.423284 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:31.423480 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:31.424015 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:55:31.424023 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:55:31.424374 (+ 351us) client_negotiation.cc:769] Sending connection context
0620 02:55:31.424634 (+ 260us) client_negotiation.cc:240] Negotiation successful
0620 02:55:31.425188 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":306,"threads_started":1}
I20260620 02:55:31.428254 15150 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45553
I20260620 02:55:31.428640 15150 heartbeater.cc:464] Registering TS with master...
I20260620 02:55:31.429399 15150 heartbeater.cc:511] Master 127.0.97.62:45553 requested a full tablet report, sending...
I20260620 02:55:31.431950 14833 ts_manager.cc:194] Registered new tserver with Master: 0460c393c90f4abd993a05c6fe147550 (127.0.97.2:38251)
I20260620 02:55:31.433321 14833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56243
W20260620 02:55:31.753685 15153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:55:31.754017 15153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:55:31.754118 15153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:55:31.761505 15153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:55:31.761619 15153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:55:31.761732 15153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:55:31.771814 15153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:46245
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=36497
--tserver_master_addrs=127.0.97.62:45553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:55:31.773039 15153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:55:31.774277 15153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:55:31.788074 15161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:31.788092 15158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:31.788223 15159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:55:31.789456 15153 server_base.cc:1034] running on GCE node
I20260620 02:55:31.790166 15153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:55:31.791296 15153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:55:31.792600 15153 hybrid_clock.cc:648] HybridClock initialized: now 1781924131792549 us; error 42 us; skew 500 ppm
I20260620 02:55:31.792961 15153 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:55:31 dist-test-slave-8p8c krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:55:31.796780 15153 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:55:31.798450 15153 webserver.cc:460] Webserver started at http://127.0.97.3:36497/ using document root <none> and password file <none>
I20260620 02:55:31.799167 15153 fs_manager.cc:359] Metadata directory not provided
I20260620 02:55:31.799328 15153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:55:31.804729 15153 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 02:55:31.807962 15168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:31.809516 15153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 02:55:31.809716 15153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/wal
uuid: "6494b6ffa1464d9cb2f168f618805049"
format_stamp: "Formatted at 2026-06-20 02:55:28 on dist-test-slave-8p8c"
server_key: "bf7f702fe647445822cff42af22bcfde"
server_key_iv: "ac93eb6ff9762522ae5c9e7e5f1b6478"
server_key_version: "encryptionkey@0"
I20260620 02:55:31.810767 15153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:55:31.822507 15153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:55:31.826823 15153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:55:31.827106 15153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:55:31.828039 15153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:55:31.829370 15153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:55:31.829499 15153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:31.829628 15153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:55:31.829725 15153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:31.866569 15153 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46245
I20260620 02:55:31.866706 15280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46245 every 8 connection(s)
I20260620 02:55:31.868260 15153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:55:31.877447 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 15153
Jun 20 02:55:31 dist-test-slave-8p8c krb5kdc[14249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:55:31.890650 15020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:31.870918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45669 (local address 127.0.97.62:45553)
0620 02:55:31.871143 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:31.871150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:31.872325 (+ 1175us) server_negotiation.cc:407] Connection header received
0620 02:55:31.874148 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:31.874152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:31.874220 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:31.874309 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:31.876068 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:31.876794 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:31.878847 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:31.879696 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:31.879897 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:31.884114 (+ 4217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:31.884119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:31.884123 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:31.884156 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:31.887200 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:31.887958 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:31.887962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:31.887965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:31.888186 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:31.888626 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:31.888633 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:31.888637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:31.889108 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:31.889272 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:31.890033 (+ 761us) server_negotiation.cc:299] Negotiation successful
0620 02:55:31.890307 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 02:55:31.891817 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:31.871427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.3:45669)
0620 02:55:31.872168 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:31.872206 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:31.873880 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:31.874604 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:31.874612 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:31.875110 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:55:31.875688 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:31.875704 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:31.876973 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:31.876977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:31.878625 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:31.878644 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:31.879964 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:31.879973 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:31.880237 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:31.881002 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:55:31.881026 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:55:31.883877 (+ 2851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:55:31.887345 (+ 3468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:31.887355 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:31.887373 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:31.887807 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:31.888318 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:31.888321 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:31.888324 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:31.888504 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:31.889265 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:55:31.889272 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:55:31.889623 (+ 351us) client_negotiation.cc:769] Sending connection context
0620 02:55:31.889869 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 02:55:31.890306 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":301,"threads_started":1}
I20260620 02:55:31.893363 15284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:31.878483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49488 (local address 127.0.97.62:45553)
0620 02:55:31.879311 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:31.879317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:31.879506 (+ 189us) server_negotiation.cc:407] Connection header received
0620 02:55:31.879721 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:31.879724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:31.879793 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:31.879913 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:31.881584 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:31.882423 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:31.883965 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:31.884491 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:31.884729 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:31.887343 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:31.887349 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:31.887352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:31.887383 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:31.890318 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:31.891166 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:31.891170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:31.891173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:31.891303 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:31.891757 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:31.891761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:31.891764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:31.892183 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:31.892458 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:31.892763 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 02:55:31.893003 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":440,"threads_started":1}
I20260620 02:55:31.893527 15281 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45553
I20260620 02:55:31.893890 15281 heartbeater.cc:464] Registering TS with master...
I20260620 02:55:31.894745 15281 heartbeater.cc:511] Master 127.0.97.62:45553 requested a full tablet report, sending...
I20260620 02:55:31.896616 14832 ts_manager.cc:194] Registered new tserver with Master: 6494b6ffa1464d9cb2f168f618805049 (127.0.97.3:46245)
I20260620 02:55:31.897477 14832 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45669
I20260620 02:55:31.909557 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:55:31.917286 15284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:31.910861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49498 (local address 127.0.97.62:45553)
0620 02:55:31.911203 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:31.911209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:31.911487 (+ 278us) server_negotiation.cc:407] Connection header received
0620 02:55:31.911743 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:31.911747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:31.911893 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:31.912083 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:55:31.912518 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:31.913245 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:31.915038 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:31.915478 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:31.915617 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:31.916141 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:55:31.916685 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:55:31.916862 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:31.916913 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 02:55:31.916995 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260620 02:55:31.918483 14832 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:49498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:55:31.925918 14832 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:55:31.963766 15293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:31.939029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48340 (local address 127.0.97.3:46245)
0620 02:55:31.953752 (+ 14723us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:31.953766 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:31.953813 (+ 47us) server_negotiation.cc:407] Connection header received
0620 02:55:31.954514 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:31.954525 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:31.954850 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:31.955905 (+ 1055us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:55:31.956220 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:31.957828 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:31.960921 (+ 3093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:31.962311 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:31.962508 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:31.962756 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:31.963140 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 02:55:31.963317 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14511,"thread_start_us":14339,"threads_started":1}
I20260620 02:55:31.963812 15291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:31.938522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38251 (local address 127.0.0.1:49706)
0620 02:55:31.939618 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:31.939673 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:31.939896 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:31.957069 (+ 17173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:31.957077 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:31.957119 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:55:31.957548 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:31.957571 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:31.959229 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:31.959237 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:31.960822 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:31.960836 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:31.962828 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:31.962838 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:31.963044 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:31.963082 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 02:55:31.963140 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 02:55:31.963355 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":75,"thread_start_us":1360,"threads_started":1}
I20260620 02:55:31.963865 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:31.938529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46245 (local address 127.0.0.1:48340)
0620 02:55:31.952576 (+ 14047us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:31.952603 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:31.952764 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:31.955113 (+ 2349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:31.955121 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:31.955157 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:55:31.955647 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:31.955664 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:31.958010 (+ 2346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:31.958017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:31.960661 (+ 2644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:31.960681 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:31.962590 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:31.962597 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:31.962887 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:31.962928 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 02:55:31.963099 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 02:55:31.963365 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13869,"mutex_wait_us":868,"thread_start_us":14878,"threads_started":1}
I20260620 02:55:31.964843 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:31.938760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49706 (local address 127.0.97.2:38251)
0620 02:55:31.956412 (+ 17652us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:31.956424 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:31.956454 (+ 30us) server_negotiation.cc:407] Connection header received
0620 02:55:31.956552 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:31.956559 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:31.956831 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:31.957038 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:55:31.957749 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:31.959087 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:31.961037 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:31.962556 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:31.964288 (+ 1732us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:31.964432 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:31.964494 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 02:55:31.964611 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17481,"thread_start_us":17352,"threads_started":1}
I20260620 02:55:31.965166 15018 heartbeater.cc:503] Master 127.0.97.62:45553 was elected leader, sending a full tablet report...
I20260620 02:55:31.967490 15296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:31.941214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58090 (local address 127.0.97.1:43863)
0620 02:55:31.957558 (+ 16344us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:31.957566 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:31.957591 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:55:31.957674 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:31.957681 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:31.957918 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:31.958126 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:55:31.958873 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:31.960444 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:31.963069 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:31.965178 (+ 2109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:31.965311 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:31.965538 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:31.966951 (+ 1413us) server_negotiation.cc:299] Negotiation successful
0620 02:55:31.967187 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16181,"thread_start_us":16050,"threads_started":1}
I20260620 02:55:31.968465 15216 tablet_service.cc:1467] Processing CreateTablet for tablet 9737c3ac517a4952ba55edf2b335e6a4 (DEFAULT_TABLE table=test-table [id=8641e9841bd64fdd8c70f71e1ff0ad48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:55:31.969308 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:31.940776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43863 (local address 127.0.0.1:58090)
0620 02:55:31.954525 (+ 13749us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:31.954544 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:31.954652 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:31.958135 (+ 3483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:31.958140 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:31.958170 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:55:31.958631 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:31.958646 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:31.960729 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:31.960734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:31.962902 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:31.962913 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:31.965690 (+ 2777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:31.965696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:31.965868 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:31.965915 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 02:55:31.965994 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 02:55:31.966138 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13616,"mutex_wait_us":23,"thread_start_us":12091,"threads_started":1}
I20260620 02:55:31.969106 15085 tablet_service.cc:1467] Processing CreateTablet for tablet 9737c3ac517a4952ba55edf2b335e6a4 (DEFAULT_TABLE table=test-table [id=8641e9841bd64fdd8c70f71e1ff0ad48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:55:31.970220 15216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9737c3ac517a4952ba55edf2b335e6a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:55:31.970484 15085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9737c3ac517a4952ba55edf2b335e6a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:55:31.973173 14953 tablet_service.cc:1467] Processing CreateTablet for tablet 9737c3ac517a4952ba55edf2b335e6a4 (DEFAULT_TABLE table=test-table [id=8641e9841bd64fdd8c70f71e1ff0ad48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:55:31.974953 14953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9737c3ac517a4952ba55edf2b335e6a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:55:31.984517 15297 tablet_bootstrap.cc:492] T 9737c3ac517a4952ba55edf2b335e6a4 P 6494b6ffa1464d9cb2f168f618805049: Bootstrap starting.
I20260620 02:55:31.987088 15298 tablet_bootstrap.cc:492] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550: Bootstrap starting.
I20260620 02:55:31.989818 15297 tablet_bootstrap.cc:654] T 9737c3ac517a4952ba55edf2b335e6a4 P 6494b6ffa1464d9cb2f168f618805049: Neither blocks nor log segments found. Creating new log.
I20260620 02:55:31.990906 15298 tablet_bootstrap.cc:654] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550: Neither blocks nor log segments found. Creating new log.
I20260620 02:55:31.991051 15297 log.cc:824] T 9737c3ac517a4952ba55edf2b335e6a4 P 6494b6ffa1464d9cb2f168f618805049: Log is configured to *not* fsync() on all Append() calls
I20260620 02:55:31.992463 15298 log.cc:824] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550: Log is configured to *not* fsync() on all Append() calls
I20260620 02:55:31.993367 15300 tablet_bootstrap.cc:492] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6: Bootstrap starting.
I20260620 02:55:31.994333 15297 tablet_bootstrap.cc:492] T 9737c3ac517a4952ba55edf2b335e6a4 P 6494b6ffa1464d9cb2f168f618805049: No bootstrap required, opened a new log
I20260620 02:55:31.994695 15297 ts_tablet_manager.cc:1397] T 9737c3ac517a4952ba55edf2b335e6a4 P 6494b6ffa1464d9cb2f168f618805049: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260620 02:55:31.996501 15298 tablet_bootstrap.cc:492] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550: No bootstrap required, opened a new log
I20260620 02:55:31.996810 15298 ts_tablet_manager.cc:1397] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 02:55:31.997839 15300 tablet_bootstrap.cc:654] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6: Neither blocks nor log segments found. Creating new log.
I20260620 02:55:31.999174 15300 log.cc:824] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6: Log is configured to *not* fsync() on all Append() calls
I20260620 02:55:32.003506 15300 tablet_bootstrap.cc:492] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6: No bootstrap required, opened a new log
I20260620 02:55:32.003811 15300 ts_tablet_manager.cc:1397] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260620 02:55:32.005486 15298 raft_consensus.cc:348] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0460c393c90f4abd993a05c6fe147550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38251 } } peers { permanent_uuid: "6494b6ffa1464d9cb2f168f618805049" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46245 } } peers { permanent_uuid: "bebde3ed8c2b454495b46420279facf6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } }
I20260620 02:55:32.005882 15298 raft_consensus.cc:374] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:55:32.006018 15298 raft_consensus.cc:729] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0460c393c90f4abd993a05c6fe147550, State: Initialized, Role: FOLLOWER
I20260620 02:55:32.005771 15297 raft_consensus.cc:348] T 9737c3ac517a4952ba55edf2b335e6a4 P 6494b6ffa1464d9cb2f168f618805049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0460c393c90f4abd993a05c6fe147550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38251 } } peers { permanent_uuid: "6494b6ffa1464d9cb2f168f618805049" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46245 } } peers { permanent_uuid: "bebde3ed8c2b454495b46420279facf6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } }
I20260620 02:55:32.006230 15297 raft_consensus.cc:374] T 9737c3ac517a4952ba55edf2b335e6a4 P 6494b6ffa1464d9cb2f168f618805049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:55:32.006421 15297 raft_consensus.cc:729] T 9737c3ac517a4952ba55edf2b335e6a4 P 6494b6ffa1464d9cb2f168f618805049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6494b6ffa1464d9cb2f168f618805049, State: Initialized, Role: FOLLOWER
I20260620 02:55:32.006749 15298 consensus_queue.cc:260] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0460c393c90f4abd993a05c6fe147550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38251 } } peers { permanent_uuid: "6494b6ffa1464d9cb2f168f618805049" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46245 } } peers { permanent_uuid: "bebde3ed8c2b454495b46420279facf6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } }
I20260620 02:55:32.007033 15297 consensus_queue.cc:260] T 9737c3ac517a4952ba55edf2b335e6a4 P 6494b6ffa1464d9cb2f168f618805049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0460c393c90f4abd993a05c6fe147550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38251 } } peers { permanent_uuid: "6494b6ffa1464d9cb2f168f618805049" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46245 } } peers { permanent_uuid: "bebde3ed8c2b454495b46420279facf6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } }
I20260620 02:55:32.008028 15150 heartbeater.cc:503] Master 127.0.97.62:45553 was elected leader, sending a full tablet report...
I20260620 02:55:32.008507 15298 ts_tablet_manager.cc:1428] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260620 02:55:32.013007 15281 heartbeater.cc:503] Master 127.0.97.62:45553 was elected leader, sending a full tablet report...
I20260620 02:55:32.014176 15297 ts_tablet_manager.cc:1428] T 9737c3ac517a4952ba55edf2b335e6a4 P 6494b6ffa1464d9cb2f168f618805049: Time spent starting tablet: real 0.019s user 0.018s sys 0.000s
I20260620 02:55:32.018492 15300 raft_consensus.cc:348] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0460c393c90f4abd993a05c6fe147550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38251 } } peers { permanent_uuid: "6494b6ffa1464d9cb2f168f618805049" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46245 } } peers { permanent_uuid: "bebde3ed8c2b454495b46420279facf6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } }
I20260620 02:55:32.018882 15300 raft_consensus.cc:374] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:55:32.018993 15300 raft_consensus.cc:729] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bebde3ed8c2b454495b46420279facf6, State: Initialized, Role: FOLLOWER
I20260620 02:55:32.019549 15300 consensus_queue.cc:260] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0460c393c90f4abd993a05c6fe147550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38251 } } peers { permanent_uuid: "6494b6ffa1464d9cb2f168f618805049" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46245 } } peers { permanent_uuid: "bebde3ed8c2b454495b46420279facf6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } }
I20260620 02:55:32.021080 15300 ts_tablet_manager.cc:1428] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
W20260620 02:55:32.119936 15282 tablet.cc:2369] T 9737c3ac517a4952ba55edf2b335e6a4 P 6494b6ffa1464d9cb2f168f618805049: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:55:32.136272 15303 raft_consensus.cc:482] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:55:32.136564 15303 raft_consensus.cc:504] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0460c393c90f4abd993a05c6fe147550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38251 } } peers { permanent_uuid: "6494b6ffa1464d9cb2f168f618805049" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46245 } } peers { permanent_uuid: "bebde3ed8c2b454495b46420279facf6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } }
I20260620 02:55:32.138492 15303 leader_election.cc:288] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6494b6ffa1464d9cb2f168f618805049 (127.0.97.3:46245), bebde3ed8c2b454495b46420279facf6 (127.0.97.1:43863)
I20260620 02:55:32.146014 15293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:32.139027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35829 (local address 127.0.97.3:46245)
0620 02:55:32.139388 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:32.139398 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:32.140405 (+ 1007us) server_negotiation.cc:407] Connection header received
0620 02:55:32.140614 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:32.140619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:32.140697 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:32.140792 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:55:32.141468 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:32.142323 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:32.144258 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:32.145026 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:32.145170 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:32.145294 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:32.145617 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 02:55:32.145749 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260620 02:55:32.146020 15296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:32.139121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47933 (local address 127.0.97.1:43863)
0620 02:55:32.139400 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:32.139409 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:32.140188 (+ 779us) server_negotiation.cc:407] Connection header received
0620 02:55:32.140395 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:32.140402 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:32.140486 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:32.140597 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:55:32.141220 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:32.142084 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:32.144254 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:32.145041 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:32.145210 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:32.145313 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:32.145655 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 02:55:32.145777 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 02:55:32.146160 15307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:32.139060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46245 (local address 127.0.97.2:35829)
0620 02:55:32.140277 (+ 1217us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:32.140315 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:32.140458 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:32.140827 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:32.140830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:32.140851 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:55:32.141282 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:32.141298 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:32.142500 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:32.142505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:32.144097 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:32.144109 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:32.145262 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:32.145268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:32.145458 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:32.145481 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 02:55:32.145562 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 02:55:32.145712 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1051,"mutex_wait_us":10,"thread_start_us":854,"threads_started":1}
I20260620 02:55:32.146405 15306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:32.138733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43863 (local address 127.0.97.2:47933)
0620 02:55:32.140054 (+ 1321us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:32.140072 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:32.140243 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:32.140613 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:32.140618 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:32.140644 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:55:32.141068 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:32.141079 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:32.142267 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:32.142273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:32.144096 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:32.144110 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:32.145314 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:32.145322 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:32.145480 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:32.145503 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 02:55:32.145623 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 02:55:32.145751 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1167,"mutex_wait_us":107,"thread_start_us":911,"threads_started":1}
I20260620 02:55:32.147166 15236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9737c3ac517a4952ba55edf2b335e6a4" candidate_uuid: "0460c393c90f4abd993a05c6fe147550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6494b6ffa1464d9cb2f168f618805049" is_pre_election: true
I20260620 02:55:32.147426 14973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9737c3ac517a4952ba55edf2b335e6a4" candidate_uuid: "0460c393c90f4abd993a05c6fe147550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebde3ed8c2b454495b46420279facf6" is_pre_election: true
I20260620 02:55:32.147691 15236 raft_consensus.cc:2454] T 9737c3ac517a4952ba55edf2b335e6a4 P 6494b6ffa1464d9cb2f168f618805049 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0460c393c90f4abd993a05c6fe147550 in term 0.
I20260620 02:55:32.147933 14973 raft_consensus.cc:2454] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0460c393c90f4abd993a05c6fe147550 in term 0.
I20260620 02:55:32.148437 15040 leader_election.cc:302] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0460c393c90f4abd993a05c6fe147550, 6494b6ffa1464d9cb2f168f618805049; no voters:
I20260620 02:55:32.148867 15303 raft_consensus.cc:2781] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:55:32.149044 15303 raft_consensus.cc:482] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:55:32.149122 15303 raft_consensus.cc:3037] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:55:32.151340 15303 raft_consensus.cc:504] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0460c393c90f4abd993a05c6fe147550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38251 } } peers { permanent_uuid: "6494b6ffa1464d9cb2f168f618805049" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46245 } } peers { permanent_uuid: "bebde3ed8c2b454495b46420279facf6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } }
I20260620 02:55:32.151981 15303 leader_election.cc:288] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550 [CANDIDATE]: Term 1 election: Requested vote from peers 6494b6ffa1464d9cb2f168f618805049 (127.0.97.3:46245), bebde3ed8c2b454495b46420279facf6 (127.0.97.1:43863)
W20260620 02:55:32.152122 15151 tablet.cc:2369] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:55:32.152433 15236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9737c3ac517a4952ba55edf2b335e6a4" candidate_uuid: "0460c393c90f4abd993a05c6fe147550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6494b6ffa1464d9cb2f168f618805049"
I20260620 02:55:32.152485 14973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9737c3ac517a4952ba55edf2b335e6a4" candidate_uuid: "0460c393c90f4abd993a05c6fe147550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebde3ed8c2b454495b46420279facf6"
I20260620 02:55:32.152660 15236 raft_consensus.cc:3037] T 9737c3ac517a4952ba55edf2b335e6a4 P 6494b6ffa1464d9cb2f168f618805049 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:55:32.152688 14973 raft_consensus.cc:3037] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:55:32.155104 14973 raft_consensus.cc:2454] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0460c393c90f4abd993a05c6fe147550 in term 1.
I20260620 02:55:32.155107 15236 raft_consensus.cc:2454] T 9737c3ac517a4952ba55edf2b335e6a4 P 6494b6ffa1464d9cb2f168f618805049 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0460c393c90f4abd993a05c6fe147550 in term 1.
I20260620 02:55:32.155552 15039 leader_election.cc:302] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0460c393c90f4abd993a05c6fe147550, bebde3ed8c2b454495b46420279facf6; no voters:
I20260620 02:55:32.155896 15303 raft_consensus.cc:2781] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:55:32.156340 15303 raft_consensus.cc:686] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550 [term 1 LEADER]: Becoming Leader. State: Replica: 0460c393c90f4abd993a05c6fe147550, State: Running, Role: LEADER
I20260620 02:55:32.156893 15303 consensus_queue.cc:237] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0460c393c90f4abd993a05c6fe147550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38251 } } peers { permanent_uuid: "6494b6ffa1464d9cb2f168f618805049" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46245 } } peers { permanent_uuid: "bebde3ed8c2b454495b46420279facf6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } }
I20260620 02:55:32.161847 14832 catalog_manager.cc:5495] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0460c393c90f4abd993a05c6fe147550 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "0460c393c90f4abd993a05c6fe147550" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0460c393c90f4abd993a05c6fe147550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6494b6ffa1464d9cb2f168f618805049" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bebde3ed8c2b454495b46420279facf6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } health_report { overall_health: UNKNOWN } } }
W20260620 02:55:32.177165 15019 tablet.cc:2369] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:55:32.189383 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:32.183499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50646 (local address 127.0.97.2:38251)
0620 02:55:32.183725 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:32.183732 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:32.183802 (+ 70us) server_negotiation.cc:407] Connection header received
0620 02:55:32.184059 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:32.184064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:32.184140 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:32.184295 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:55:32.184679 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:32.185567 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:32.187254 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:32.187672 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:32.187808 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:32.188282 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:55:32.188799 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:55:32.188974 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:32.189024 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 02:55:32.189119 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 02:55:32.197803 15236 raft_consensus.cc:1261] T 9737c3ac517a4952ba55edf2b335e6a4 P 6494b6ffa1464d9cb2f168f618805049 [term 1 FOLLOWER]: Refusing update from remote peer 0460c393c90f4abd993a05c6fe147550: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:55:32.197942 14973 raft_consensus.cc:1261] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [term 1 FOLLOWER]: Refusing update from remote peer 0460c393c90f4abd993a05c6fe147550: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:55:32.198823 15308 consensus_queue.cc:1035] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6494b6ffa1464d9cb2f168f618805049" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:55:32.199194 15303 consensus_queue.cc:1035] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bebde3ed8c2b454495b46420279facf6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:55:32.210239 15311 mvcc.cc:205] Tried to move back new op lower bound from 7298761245472333824 to 7298761245319946240. Current Snapshot: MvccSnapshot[applied={T|T < 7298761245472333824}]
I20260620 02:55:32.210932 15312 mvcc.cc:205] Tried to move back new op lower bound from 7298761245472333824 to 7298761245319946240. Current Snapshot: MvccSnapshot[applied={T|T < 7298761245472333824}]
I20260620 02:55:32.226032 14833 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:49498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:55:32.226327 14833 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:49498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:55:32.230304 14833 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73: Sending DeleteTablet for 3 replicas of tablet 9737c3ac517a4952ba55edf2b335e6a4
I20260620 02:55:32.231750 15216 tablet_service.cc:1514] Processing DeleteTablet for tablet 9737c3ac517a4952ba55edf2b335e6a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:55:32 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48340
I20260620 02:55:32.231750 15085 tablet_service.cc:1514] Processing DeleteTablet for tablet 9737c3ac517a4952ba55edf2b335e6a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:55:32 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49706
I20260620 02:55:32.231802 14953 tablet_service.cc:1514] Processing DeleteTablet for tablet 9737c3ac517a4952ba55edf2b335e6a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:55:32 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58090
I20260620 02:55:32.232813 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 14870
I20260620 02:55:32.258131 15296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:32.234276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44606 (local address 127.0.97.1:43863)
0620 02:55:32.234463 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:32.234468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:32.234486 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:55:32.234550 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:32.234555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:32.234639 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:32.234738 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:32.236719 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:32.237912 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:32.239964 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:32.240605 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:32.240724 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:32.248439 (+ 7715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:32.248459 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:32.248479 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:32.248533 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:32.252548 (+ 4015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:32.254099 (+ 1551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:32.254110 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:32.254114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:32.254308 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:32.255001 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:32.255017 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:32.255021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:32.255494 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:32.255781 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:32.257433 (+ 1652us) server_negotiation.cc:299] Negotiation successful
0620 02:55:32.257679 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 02:55:32.259785 15013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:55:32.278944 15321 tablet_replica.cc:330] T 9737c3ac517a4952ba55edf2b335e6a4 P 6494b6ffa1464d9cb2f168f618805049: stopping tablet replica
I20260620 02:55:32.279312 15320 tablet_replica.cc:330] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550: stopping tablet replica
I20260620 02:55:32.279614 15321 raft_consensus.cc:2229] T 9737c3ac517a4952ba55edf2b335e6a4 P 6494b6ffa1464d9cb2f168f618805049 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:55:32.279846 15320 raft_consensus.cc:2229] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:55:32.280314 15321 raft_consensus.cc:2258] T 9737c3ac517a4952ba55edf2b335e6a4 P 6494b6ffa1464d9cb2f168f618805049 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:55:32.280603 15320 raft_consensus.cc:2258] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:55:32.282804 15321 ts_tablet_manager.cc:1905] T 9737c3ac517a4952ba55edf2b335e6a4 P 6494b6ffa1464d9cb2f168f618805049: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:55:32.282970 15320 ts_tablet_manager.cc:1905] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:55:32.290701 15321 ts_tablet_manager.cc:1918] T 9737c3ac517a4952ba55edf2b335e6a4 P 6494b6ffa1464d9cb2f168f618805049: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:55:32.290975 15321 log.cc:1196] T 9737c3ac517a4952ba55edf2b335e6a4 P 6494b6ffa1464d9cb2f168f618805049: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/wal/wals/9737c3ac517a4952ba55edf2b335e6a4
I20260620 02:55:32.291190 15320 ts_tablet_manager.cc:1918] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:55:32.291447 15320 log.cc:1196] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/wal/wals/9737c3ac517a4952ba55edf2b335e6a4
I20260620 02:55:32.291716 15321 ts_tablet_manager.cc:1939] T 9737c3ac517a4952ba55edf2b335e6a4 P 6494b6ffa1464d9cb2f168f618805049: Deleting consensus metadata
I20260620 02:55:32.292097 15320 ts_tablet_manager.cc:1939] T 9737c3ac517a4952ba55edf2b335e6a4 P 0460c393c90f4abd993a05c6fe147550: Deleting consensus metadata
I20260620 02:55:32.293782 14819 catalog_manager.cc:4841] TS 6494b6ffa1464d9cb2f168f618805049 (127.0.97.3:46245): tablet 9737c3ac517a4952ba55edf2b335e6a4 (table test-table [id=8641e9841bd64fdd8c70f71e1ff0ad48]) successfully deleted
I20260620 02:55:32.293962 14818 catalog_manager.cc:4841] TS 0460c393c90f4abd993a05c6fe147550 (127.0.97.2:38251): tablet 9737c3ac517a4952ba55edf2b335e6a4 (table test-table [id=8641e9841bd64fdd8c70f71e1ff0ad48]) successfully deleted
I20260620 02:55:32.371079 15322 tablet_replica.cc:330] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6: stopping tablet replica
I20260620 02:55:32.371372 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 14870
I20260620 02:55:32.397994 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 15022
W20260620 02:55:32.402540 14818 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:43863 (error 108)
I20260620 02:55:32.415776 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:32.415226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43863 (local address 127.0.0.1:44614)
0620 02:55:32.415420 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:32.415571 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 02:55:32.416242 14818 catalog_manager.cc:4584] TS bebde3ed8c2b454495b46420279facf6 (127.0.97.1:43863): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9737c3ac517a4952ba55edf2b335e6a4: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43863: connect: Connection refused (error 111)
I20260620 02:55:32.429752 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:32.407602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50648 (local address 127.0.97.2:38251)
0620 02:55:32.407777 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:32.407783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:32.407801 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:55:32.407860 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:32.407864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:32.407938 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:32.408048 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:32.409935 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:32.411180 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:32.413102 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:32.413762 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:32.413873 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:32.421332 (+ 7459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:32.421348 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:32.421367 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:32.421424 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:32.425627 (+ 4203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:32.426591 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:32.426601 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:32.426605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:32.426786 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:32.427203 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:32.427208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:32.427211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:32.427638 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:32.427970 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:32.429138 (+ 1168us) server_negotiation.cc:299] Negotiation successful
0620 02:55:32.429354 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 02:55:32.430646 15145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:55:32.470296 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:32.469771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43863 (local address 127.0.0.1:44620)
0620 02:55:32.470036 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:32.470148 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 02:55:32.471411 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:32.471046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43863 (local address 127.0.0.1:44624)
0620 02:55:32.471206 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:32.471285 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 02:55:32.531875 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:32.531362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43863 (local address 127.0.0.1:44636)
0620 02:55:32.531594 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:32.531688 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 02:55:32.532967 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:32.532626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43863 (local address 127.0.0.1:44642)
0620 02:55:32.532779 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:32.532858 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 02:55:32.563182 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 15022
I20260620 02:55:32.583384 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 15153
I20260620 02:55:32.597486 15293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:32.584181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33522 (local address 127.0.97.3:46245)
0620 02:55:32.584369 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:32.584374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:32.584563 (+ 189us) server_negotiation.cc:407] Connection header received
0620 02:55:32.584769 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:32.584774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:32.584865 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:32.584978 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:32.586673 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:32.587424 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:32.588797 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:32.589198 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:32.589302 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:32.591965 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:32.591979 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:32.591996 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:32.592064 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:32.594840 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:32.595515 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:32.595524 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:32.595526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:32.595691 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:32.596059 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:32.596063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:32.596065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:32.596444 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:32.596684 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:32.597044 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 02:55:32.597213 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 02:55:32.598182 15276 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:55:32.613344 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:32.612873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43863 (local address 127.0.0.1:44644)
0620 02:55:32.613075 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:32.613180 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 02:55:32.614337 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:32.613958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43863 (local address 127.0.0.1:44646)
0620 02:55:32.614159 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:32.614226 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260620 02:55:32.688488 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 15153
I20260620 02:55:32.705724 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 14800
I20260620 02:55:32.706984 14863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:55:32.825136 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:32.824425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43863 (local address 127.0.0.1:44650)
0620 02:55:32.824760 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:32.824879 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 02:55:32.825176 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 14800
I20260620 02:55:32.848021 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45553
--webserver_interface=127.0.97.62
--webserver_port=37747
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45553
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:55:33.190850 15328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:55:33.191154 15328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:55:33.191246 15328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:55:33.199007 15328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:55:33.199151 15328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:55:33.199201 15328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:55:33.199262 15328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:55:33.199306 15328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:55:33.209497 15328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=37747
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:55:33.211004 15328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:55:33.212307 15328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:55:33.224666 15333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:33.224684 15334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:33.224701 15336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:55:33.225631 15328 server_base.cc:1034] running on GCE node
I20260620 02:55:33.226598 15328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:55:33.228133 15328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:55:33.229399 15328 hybrid_clock.cc:648] HybridClock initialized: now 1781924133229278 us; error 108 us; skew 500 ppm
I20260620 02:55:33.229882 15328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:55:33 dist-test-slave-8p8c krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:55:33.233177 15328 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:55:33.234835 15328 webserver.cc:460] Webserver started at http://127.0.97.62:37747/ using document root <none> and password file <none>
I20260620 02:55:33.235509 15328 fs_manager.cc:359] Metadata directory not provided
I20260620 02:55:33.235618 15328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:55:33.241171 15328 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 02:55:33.244390 15343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:33.245968 15328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:55:33.246167 15328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "9bf053f892fc4f9689fbb7cbabc2fc73"
format_stamp: "Formatted at 2026-06-20 02:55:27 on dist-test-slave-8p8c"
server_key: "257a01dbdb3f862d1684bece3c79d91e"
server_key_iv: "595294467f5b1a5b124125b75cdcf239"
server_key_version: "encryptionkey@0"
I20260620 02:55:33.247265 15328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:55:33.267410 15328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:55:33.271991 15328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:55:33.272337 15328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:55:33.292511 15328 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45553
I20260620 02:55:33.292533 15394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45553 every 8 connection(s)
I20260620 02:55:33.294011 15328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:55:33.300869 15395 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:55:33.302995 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 15328
I20260620 02:55:33.303793 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:43863
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45553
--webserver_port=35225
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:55:33.304276 15395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73: Bootstrap starting.
I20260620 02:55:33.331878 15395 log.cc:824] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73: Log is configured to *not* fsync() on all Append() calls
I20260620 02:55:33.348388 15395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:55:33.348950 15395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73: Bootstrap complete.
I20260620 02:55:33.358592 15395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf053f892fc4f9689fbb7cbabc2fc73" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45553 } }
I20260620 02:55:33.359525 15395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bf053f892fc4f9689fbb7cbabc2fc73, State: Initialized, Role: FOLLOWER
I20260620 02:55:33.360232 15395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf053f892fc4f9689fbb7cbabc2fc73" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45553 } }
I20260620 02:55:33.360476 15395 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:55:33.360597 15395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:55:33.360795 15395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [term 2 FOLLOWER]: Advancing to term 3
I20260620 02:55:33.364791 15395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf053f892fc4f9689fbb7cbabc2fc73" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45553 } }
I20260620 02:55:33.365301 15395 leader_election.cc:302] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bf053f892fc4f9689fbb7cbabc2fc73; no voters:
I20260620 02:55:33.366037 15395 leader_election.cc:288] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 02:55:33.366168 15399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 02:55:33.367403 15399 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [term 3 LEADER]: Becoming Leader. State: Replica: 9bf053f892fc4f9689fbb7cbabc2fc73, State: Running, Role: LEADER
I20260620 02:55:33.368026 15399 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf053f892fc4f9689fbb7cbabc2fc73" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45553 } }
I20260620 02:55:33.368355 15395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:55:33.370620 15401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bf053f892fc4f9689fbb7cbabc2fc73. Latest consensus state: current_term: 3 leader_uuid: "9bf053f892fc4f9689fbb7cbabc2fc73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf053f892fc4f9689fbb7cbabc2fc73" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45553 } } }
I20260620 02:55:33.370977 15401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [sys.catalog]: This master's current role is: LEADER
I20260620 02:55:33.371356 15400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9bf053f892fc4f9689fbb7cbabc2fc73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf053f892fc4f9689fbb7cbabc2fc73" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45553 } } }
I20260620 02:55:33.371743 15400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bf053f892fc4f9689fbb7cbabc2fc73 [sys.catalog]: This master's current role is: LEADER
I20260620 02:55:33.382819 15406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:55:33.392100 15406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:55:33.393664 15406 catalog_manager.cc:1259] Loaded cluster ID: 2316868da5294711abf39c1bfe59ff46
I20260620 02:55:33.393821 15406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 02:55:33.396080 15406 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb4b4ef79ad google::LogMessage::Fail() at ??:0
@ 0x7fb4b4efa88a google::LogMessage::SendToLog() at ??:0
@ 0x7fb4b4ef74de google::LogMessage::Flush() at ??:0
@ 0x7fb4b4ef8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb4ca92a512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fb4ca927976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb4ca9af610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fb4ca9af439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb4b7b43345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb4b7b4691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb4b7b46799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb4b7b21eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb4badfb6db start_thread at ??:0
@ 0x7fb4b3d9161f clone at ??:0
W20260620 02:55:33.717574 15397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:55:33.717845 15397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:55:33.717939 15397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:55:33.725193 15397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:55:33.725296 15397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:55:33.725406 15397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:55:33.735175 15397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:43863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=35225
--tserver_master_addrs=127.0.97.62:45553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:55:33.736389 15397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:55:33.737664 15397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:55:33.751081 15436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:33.751150 15439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:33.751235 15437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:55:33.752501 15397 server_base.cc:1034] running on GCE node
I20260620 02:55:33.753168 15397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:55:33.754200 15397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:55:33.755474 15397 hybrid_clock.cc:648] HybridClock initialized: now 1781924133755435 us; error 57 us; skew 500 ppm
I20260620 02:55:33.755903 15397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:55:33 dist-test-slave-8p8c krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:55:33.759645 15397 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:55:33.761133 15397 webserver.cc:460] Webserver started at http://127.0.97.1:35225/ using document root <none> and password file <none>
I20260620 02:55:33.761842 15397 fs_manager.cc:359] Metadata directory not provided
I20260620 02:55:33.761991 15397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:55:33.767578 15397 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:55:33.770728 15446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:33.772318 15397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 02:55:33.772502 15397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/wal
uuid: "bebde3ed8c2b454495b46420279facf6"
format_stamp: "Formatted at 2026-06-20 02:55:27 on dist-test-slave-8p8c"
server_key: "bc2d1b09ff5e735d0c5af6bc5c626116"
server_key_iv: "ebd93faa9adea0754200db325bee04db"
server_key_version: "encryptionkey@0"
I20260620 02:55:33.773501 15397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:55:33.787856 15397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:55:33.792357 15397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:55:33.792657 15397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:55:33.793726 15397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:55:33.795775 15453 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260620 02:55:33.801556 15397 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 02:55:33.801728 15397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260620 02:55:33.801877 15397 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 02:55:33.805056 15397 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 02:55:33.805161 15397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260620 02:55:33.805516 15453 tablet_bootstrap.cc:492] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6: Bootstrap starting.
I20260620 02:55:33.833815 15453 log.cc:824] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6: Log is configured to *not* fsync() on all Append() calls
I20260620 02:55:33.844655 15397 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43863
I20260620 02:55:33.844707 15560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43863 every 8 connection(s)
I20260620 02:55:33.845870 15397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:55:33.846972 15453 tablet_bootstrap.cc:492] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:55:33.847810 15453 tablet_bootstrap.cc:492] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6: Bootstrap complete.
I20260620 02:55:33.849130 15453 ts_tablet_manager.cc:1397] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6: Time spent bootstrapping tablet: real 0.044s user 0.021s sys 0.020s
I20260620 02:55:33.851089 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:33.848715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.1:47011)
0620 02:55:33.849650 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:33.849890 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":378,"threads_started":1}
I20260620 02:55:33.851347 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 15397
W20260620 02:55:33.851847 15561 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45553 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:45553: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
I20260620 02:55:33.852113 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:38251
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45553
--webserver_port=38075
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:55:33.853068 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:33.852628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.1:58443)
0620 02:55:33.852827 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:33.852918 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 02:55:33.854151 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:33.853744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.1:47743)
0620 02:55:33.853895 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:33.853994 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 02:55:33.854418 15561 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 02:55:33.857308 15453 raft_consensus.cc:348] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0460c393c90f4abd993a05c6fe147550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38251 } } peers { permanent_uuid: "6494b6ffa1464d9cb2f168f618805049" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46245 } } peers { permanent_uuid: "bebde3ed8c2b454495b46420279facf6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } }
I20260620 02:55:33.858174 15453 raft_consensus.cc:729] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bebde3ed8c2b454495b46420279facf6, State: Initialized, Role: FOLLOWER
I20260620 02:55:33.858870 15453 consensus_queue.cc:260] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0460c393c90f4abd993a05c6fe147550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38251 } } peers { permanent_uuid: "6494b6ffa1464d9cb2f168f618805049" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46245 } } peers { permanent_uuid: "bebde3ed8c2b454495b46420279facf6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } }
I20260620 02:55:33.860387 15453 ts_tablet_manager.cc:1428] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260620 02:55:33.860535 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:33.860128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.1:43521)
0620 02:55:33.860301 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:33.860393 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 02:55:33.861353 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:33.861066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.1:36233)
0620 02:55:33.861194 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:33.861253 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 02:55:34.199052 15564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:55:34.199383 15564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:55:34.199553 15564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:55:34.206777 15564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:55:34.206936 15564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:55:34.207109 15564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:55:34.216735 15564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:38251
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=38075
--tserver_master_addrs=127.0.97.62:45553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:55:34.218051 15564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:55:34.219513 15564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:55:34.233280 15571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:34.233340 15573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:34.233356 15570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:55:34.234508 15564 server_base.cc:1034] running on GCE node
I20260620 02:55:34.235172 15564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:55:34.236191 15564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:55:34.237454 15564 hybrid_clock.cc:648] HybridClock initialized: now 1781924134237393 us; error 41 us; skew 500 ppm
I20260620 02:55:34.237793 15564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:55:34 dist-test-slave-8p8c krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:55:34.241179 15564 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:55:34.242820 15564 webserver.cc:460] Webserver started at http://127.0.97.2:38075/ using document root <none> and password file <none>
I20260620 02:55:34.243527 15564 fs_manager.cc:359] Metadata directory not provided
I20260620 02:55:34.243680 15564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:55:34.249214 15564 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260620 02:55:34.252187 15580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:34.253679 15564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:55:34.253870 15564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/wal
uuid: "0460c393c90f4abd993a05c6fe147550"
format_stamp: "Formatted at 2026-06-20 02:55:28 on dist-test-slave-8p8c"
server_key: "f945092c352ae6a3a3ded08bc13a83b3"
server_key_iv: "677ebaa8a80fdb52c177e647a71d42dc"
server_key_version: "encryptionkey@0"
I20260620 02:55:34.255038 15564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:55:34.269112 15564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:55:34.274025 15564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:55:34.274263 15564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:55:34.275315 15564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:55:34.277022 15564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:55:34.277163 15564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:34.277329 15564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:55:34.277420 15564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:34.310755 15564 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38251
I20260620 02:55:34.310858 15692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38251 every 8 connection(s)
I20260620 02:55:34.312299 15564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:55:34.316774 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 15564
I20260620 02:55:34.317580 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:46245
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45553
--webserver_port=36497
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:55:34.317698 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:34.315359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.2:33365)
0620 02:55:34.316139 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:34.316389 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":331,"threads_started":1}
W20260620 02:55:34.318415 15693 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45553 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:45553: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
I20260620 02:55:34.319423 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:34.318982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.2:44315)
0620 02:55:34.319177 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:34.319272 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 02:55:34.320518 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:34.320048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.2:41249)
0620 02:55:34.320254 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:34.320353 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260620 02:55:34.320820 15693 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 02:55:34.653717 15696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:55:34.654021 15696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:55:34.654170 15696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:55:34.661774 15696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:55:34.661893 15696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:55:34.662099 15696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:55:34.672153 15696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:46245
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=36497
--tserver_master_addrs=127.0.97.62:45553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:55:34.673496 15696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:55:34.674901 15696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:55:34.688551 15701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:34.688505 15702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:34.688535 15704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:55:34.689025 15696 server_base.cc:1034] running on GCE node
I20260620 02:55:34.689846 15696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:55:34.690907 15696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:55:34.692193 15696 hybrid_clock.cc:648] HybridClock initialized: now 1781924134692135 us; error 49 us; skew 500 ppm
I20260620 02:55:34.692541 15696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:55:34 dist-test-slave-8p8c krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:55:34.695998 15696 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:55:34.697548 15696 webserver.cc:460] Webserver started at http://127.0.97.3:36497/ using document root <none> and password file <none>
I20260620 02:55:34.698225 15696 fs_manager.cc:359] Metadata directory not provided
I20260620 02:55:34.698338 15696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:55:34.703971 15696 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:55:34.706775 15711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:34.708117 15696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 02:55:34.708293 15696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/wal
uuid: "6494b6ffa1464d9cb2f168f618805049"
format_stamp: "Formatted at 2026-06-20 02:55:28 on dist-test-slave-8p8c"
server_key: "bf7f702fe647445822cff42af22bcfde"
server_key_iv: "ac93eb6ff9762522ae5c9e7e5f1b6478"
server_key_version: "encryptionkey@0"
I20260620 02:55:34.709329 15696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:55:34.724967 15696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:55:34.729149 15696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:55:34.729420 15696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:55:34.730728 15696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:55:34.732086 15696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:55:34.732224 15696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:34.732365 15696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:55:34.732460 15696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:34.766908 15696 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46245
I20260620 02:55:34.766997 15823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46245 every 8 connection(s)
I20260620 02:55:34.768390 15696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:55:34.771921 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 15696
I20260620 02:55:34.773661 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:34.771382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.3:38315)
0620 02:55:34.772242 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:34.772420 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":332,"threads_started":1}
W20260620 02:55:34.774354 15824 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45553 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:45553: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
I20260620 02:55:34.775187 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:34.774867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.3:60657)
0620 02:55:34.775012 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:34.775088 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 02:55:34.775969 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:34.775684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.3:46335)
0620 02:55:34.775801 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:34.775869 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 02:55:34.776218 15824 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 02:55:34.863183 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:34.862290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.1:51987)
0620 02:55:34.862871 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:34.862973 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":340,"threads_started":1}
I20260620 02:55:35.023741 15829 raft_consensus.cc:482] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:55:35.024242 15829 raft_consensus.cc:504] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0460c393c90f4abd993a05c6fe147550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38251 } } peers { permanent_uuid: "6494b6ffa1464d9cb2f168f618805049" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46245 } } peers { permanent_uuid: "bebde3ed8c2b454495b46420279facf6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } }
I20260620 02:55:35.025866 15829 leader_election.cc:288] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0460c393c90f4abd993a05c6fe147550 (127.0.97.2:38251), 6494b6ffa1464d9cb2f168f618805049 (127.0.97.3:46245)
Jun 20 02:55:35 dist-test-slave-8p8c krb5kdc[14249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
Jun 20 02:55:35 dist-test-slave-8p8c krb5kdc[14249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:55:35.054908 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:35.026107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38251 (local address 127.0.97.1:43807)
0620 02:55:35.026488 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:35.026543 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:35.027234 (+ 691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:35.031713 (+ 4479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:35.031721 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:35.032367 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:55:35.033365 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:35.033394 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:35.035785 (+ 2391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:35.035789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:35.038539 (+ 2750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:35.038563 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:35.040087 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:35.040093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:35.040479 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:35.041596 (+ 1117us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:55:35.041625 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:55:35.045081 (+ 3456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:55:35.050005 (+ 4924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:35.050015 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:35.050047 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:35.050955 (+ 908us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:35.051511 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:35.051517 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:35.051520 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:35.051800 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:35.052875 (+ 1075us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:55:35.052884 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:55:35.053367 (+ 483us) client_negotiation.cc:769] Sending connection context
0620 02:55:35.053792 (+ 425us) client_negotiation.cc:240] Negotiation successful
0620 02:55:35.054373 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":415,"spinlock_wait_cycles":40320}
I20260620 02:55:35.054981 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:35.026532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43807 (local address 127.0.97.2:38251)
0620 02:55:35.028128 (+ 1596us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:35.028148 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:35.028205 (+ 57us) server_negotiation.cc:407] Connection header received
0620 02:55:35.030601 (+ 2396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:35.030615 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:35.031165 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:35.031551 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:35.033627 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:35.035562 (+ 1935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:35.038737 (+ 3175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:35.039787 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:35.039996 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:35.045481 (+ 5485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:35.045494 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:35.045519 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:35.045590 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:35.049811 (+ 4221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:35.051122 (+ 1311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:35.051132 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:35.051142 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:35.051380 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:35.051930 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:35.051935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:35.051938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:35.052684 (+ 746us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:35.052872 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:35.054084 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0620 02:55:35.054545 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1355,"thread_start_us":1207,"threads_started":1}
I20260620 02:55:35.054981 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:35.026647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33179 (local address 127.0.97.3:46245)
0620 02:55:35.029539 (+ 2892us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:35.029556 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:35.029598 (+ 42us) server_negotiation.cc:407] Connection header received
0620 02:55:35.031745 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:35.031756 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:35.032179 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:35.032456 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:35.033626 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:35.035552 (+ 1926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:35.038735 (+ 3183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:35.039786 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:35.039986 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:35.045581 (+ 5595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:35.045590 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:35.045604 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:35.045664 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:35.049811 (+ 4147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:35.051122 (+ 1311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:35.051132 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:35.051142 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:35.051380 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:35.051930 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:35.051934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:35.051938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:35.052706 (+ 768us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:35.052872 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:35.054084 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0620 02:55:35.054568 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2689,"thread_start_us":2376,"threads_started":1}
I20260620 02:55:35.055488 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:35.026228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46245 (local address 127.0.97.1:33179)
0620 02:55:35.029074 (+ 2846us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:35.029095 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:35.029225 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:35.032455 (+ 3230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:35.032461 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:35.033075 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:55:35.033435 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:35.033444 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:35.035783 (+ 2339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:35.035789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:35.038539 (+ 2750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:35.038563 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:35.040062 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:35.040070 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:35.040485 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:35.041593 (+ 1108us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:55:35.041672 (+ 79us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:55:35.045290 (+ 3618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:55:35.050005 (+ 4715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:35.050015 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:35.050035 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:35.050955 (+ 920us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:35.051511 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:35.051516 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:35.051520 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:35.051800 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:35.052875 (+ 1075us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:55:35.052884 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:55:35.053368 (+ 484us) client_negotiation.cc:769] Sending connection context
0620 02:55:35.053792 (+ 424us) client_negotiation.cc:240] Negotiation successful
0620 02:55:35.054531 (+ 739us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2672,"mutex_wait_us":748,"spinlock_wait_cycles":38016,"thread_start_us":2443,"threads_started":1}
W20260620 02:55:35.058987 15448 leader_election.cc:341] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0460c393c90f4abd993a05c6fe147550 (127.0.97.2:38251): Not found: Tablet not found: 9737c3ac517a4952ba55edf2b335e6a4
W20260620 02:55:35.059382 15449 leader_election.cc:341] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6494b6ffa1464d9cb2f168f618805049 (127.0.97.3:46245): Not found: Tablet not found: 9737c3ac517a4952ba55edf2b335e6a4
I20260620 02:55:35.059562 15449 leader_election.cc:302] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bebde3ed8c2b454495b46420279facf6; no voters: 0460c393c90f4abd993a05c6fe147550, 6494b6ffa1464d9cb2f168f618805049
I20260620 02:55:35.060005 15829 raft_consensus.cc:2726] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 02:55:35.322280 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:35.321402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.2:51729)
0620 02:55:35.321896 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:35.322032 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":241,"threads_started":1}
I20260620 02:55:35.777820 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:35.776929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.3:56497)
0620 02:55:35.777406 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:35.777530 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":244,"threads_started":1}
I20260620 02:55:35.865012 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:35.864245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.1:51055)
0620 02:55:35.864762 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:35.864853 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":295,"threads_started":1}
I20260620 02:55:36.323976 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:36.323281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.2:43543)
0620 02:55:36.323734 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:36.323819 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":240,"threads_started":1}
I20260620 02:55:36.779560 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:36.778811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.3:51175)
0620 02:55:36.779293 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:36.779379 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":254,"threads_started":1}
I20260620 02:55:36.832736 15838 raft_consensus.cc:482] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:55:36.832969 15838 raft_consensus.cc:504] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0460c393c90f4abd993a05c6fe147550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38251 } } peers { permanent_uuid: "6494b6ffa1464d9cb2f168f618805049" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46245 } } peers { permanent_uuid: "bebde3ed8c2b454495b46420279facf6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } }
I20260620 02:55:36.833663 15838 leader_election.cc:288] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0460c393c90f4abd993a05c6fe147550 (127.0.97.2:38251), 6494b6ffa1464d9cb2f168f618805049 (127.0.97.3:46245)
W20260620 02:55:36.834862 15448 leader_election.cc:341] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0460c393c90f4abd993a05c6fe147550 (127.0.97.2:38251): Not found: Tablet not found: 9737c3ac517a4952ba55edf2b335e6a4
W20260620 02:55:36.835078 15449 leader_election.cc:341] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6494b6ffa1464d9cb2f168f618805049 (127.0.97.3:46245): Not found: Tablet not found: 9737c3ac517a4952ba55edf2b335e6a4
I20260620 02:55:36.835203 15449 leader_election.cc:302] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bebde3ed8c2b454495b46420279facf6; no voters: 0460c393c90f4abd993a05c6fe147550, 6494b6ffa1464d9cb2f168f618805049
I20260620 02:55:36.835490 15838 raft_consensus.cc:2726] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 02:55:36.867136 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:36.866134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.1:42229)
0620 02:55:36.866814 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:36.866911 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":318,"threads_started":1}
I20260620 02:55:37.325635 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:37.324917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.2:32993)
0620 02:55:37.325391 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:37.325469 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":258,"threads_started":1}
I20260620 02:55:37.781255 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:37.780588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.3:45279)
0620 02:55:37.781018 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:37.781100 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":232,"threads_started":1}
I20260620 02:55:37.868783 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:37.868093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.1:52411)
0620 02:55:37.868541 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:37.868627 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":231,"threads_started":1}
I20260620 02:55:38.327344 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:38.326587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.2:54169)
0620 02:55:38.327067 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:38.327150 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":227,"threads_started":1}
I20260620 02:55:38.407843 15844 raft_consensus.cc:482] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:55:38.408138 15844 raft_consensus.cc:504] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0460c393c90f4abd993a05c6fe147550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38251 } } peers { permanent_uuid: "6494b6ffa1464d9cb2f168f618805049" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46245 } } peers { permanent_uuid: "bebde3ed8c2b454495b46420279facf6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } }
I20260620 02:55:38.408806 15844 leader_election.cc:288] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0460c393c90f4abd993a05c6fe147550 (127.0.97.2:38251), 6494b6ffa1464d9cb2f168f618805049 (127.0.97.3:46245)
W20260620 02:55:38.409878 15449 leader_election.cc:341] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6494b6ffa1464d9cb2f168f618805049 (127.0.97.3:46245): Not found: Tablet not found: 9737c3ac517a4952ba55edf2b335e6a4
W20260620 02:55:38.410130 15448 leader_election.cc:341] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0460c393c90f4abd993a05c6fe147550 (127.0.97.2:38251): Not found: Tablet not found: 9737c3ac517a4952ba55edf2b335e6a4
I20260620 02:55:38.410264 15448 leader_election.cc:302] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bebde3ed8c2b454495b46420279facf6; no voters: 0460c393c90f4abd993a05c6fe147550, 6494b6ffa1464d9cb2f168f618805049
I20260620 02:55:38.410566 15844 raft_consensus.cc:2726] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 02:55:38.782944 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:38.782144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.3:44029)
0620 02:55:38.782659 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:38.782747 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":270,"threads_started":1}
I20260620 02:55:38.870800 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:38.869923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.1:47333)
0620 02:55:38.870484 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:38.870610 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":248,"threads_started":1}
I20260620 02:55:39.329149 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:39.328386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.2:52447)
0620 02:55:39.328889 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:39.328973 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":271,"threads_started":1}
I20260620 02:55:39.784745 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:39.783987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.3:36283)
0620 02:55:39.784470 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:39.784563 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":270,"threads_started":1}
I20260620 02:55:39.872480 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:39.871777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.1:48829)
0620 02:55:39.872225 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:39.872321 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":236,"threads_started":1}
I20260620 02:55:40.330979 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:40.330165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.2:36853)
0620 02:55:40.330697 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:40.330778 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":308,"threads_started":1}
I20260620 02:55:40.749706 15851 raft_consensus.cc:482] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:55:40.749878 15851 raft_consensus.cc:504] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0460c393c90f4abd993a05c6fe147550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38251 } } peers { permanent_uuid: "6494b6ffa1464d9cb2f168f618805049" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46245 } } peers { permanent_uuid: "bebde3ed8c2b454495b46420279facf6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } }
I20260620 02:55:40.750583 15851 leader_election.cc:288] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0460c393c90f4abd993a05c6fe147550 (127.0.97.2:38251), 6494b6ffa1464d9cb2f168f618805049 (127.0.97.3:46245)
W20260620 02:55:40.751410 15448 leader_election.cc:341] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0460c393c90f4abd993a05c6fe147550 (127.0.97.2:38251): Not found: Tablet not found: 9737c3ac517a4952ba55edf2b335e6a4
W20260620 02:55:40.751616 15449 leader_election.cc:341] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6494b6ffa1464d9cb2f168f618805049 (127.0.97.3:46245): Not found: Tablet not found: 9737c3ac517a4952ba55edf2b335e6a4
I20260620 02:55:40.751778 15449 leader_election.cc:302] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bebde3ed8c2b454495b46420279facf6; no voters: 0460c393c90f4abd993a05c6fe147550, 6494b6ffa1464d9cb2f168f618805049
I20260620 02:55:40.752072 15851 raft_consensus.cc:2726] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 02:55:40.786726 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:40.785652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.3:55075)
0620 02:55:40.786331 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:40.786471 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":370,"threads_started":1}
I20260620 02:55:40.874337 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:40.873520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.1:42881)
0620 02:55:40.874026 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:40.874118 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":271,"threads_started":1}
I20260620 02:55:41.332871 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:41.331995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.2:44629)
0620 02:55:41.332579 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:41.332681 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":266,"threads_started":1}
I20260620 02:55:41.788631 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:41.787726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.3:57413)
0620 02:55:41.788294 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:41.788390 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":271,"threads_started":1}
I20260620 02:55:41.876174 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:41.875438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.1:46867)
0620 02:55:41.875890 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:41.875984 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":228,"threads_started":1}
I20260620 02:55:42.334929 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:42.334042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.2:36853)
0620 02:55:42.334623 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:42.334724 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":251,"threads_started":1}
I20260620 02:55:42.790530 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:42.789695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.3:54633)
0620 02:55:42.790175 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:42.790265 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":231,"threads_started":1}
I20260620 02:55:42.878407 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:42.877321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.1:57903)
0620 02:55:42.877975 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:42.878095 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":340,"threads_started":1}
I20260620 02:55:42.900666 15860 raft_consensus.cc:482] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:55:42.900880 15860 raft_consensus.cc:504] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0460c393c90f4abd993a05c6fe147550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38251 } } peers { permanent_uuid: "6494b6ffa1464d9cb2f168f618805049" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46245 } } peers { permanent_uuid: "bebde3ed8c2b454495b46420279facf6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } }
I20260620 02:55:42.901939 15860 leader_election.cc:288] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0460c393c90f4abd993a05c6fe147550 (127.0.97.2:38251), 6494b6ffa1464d9cb2f168f618805049 (127.0.97.3:46245)
W20260620 02:55:42.903086 15448 leader_election.cc:341] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0460c393c90f4abd993a05c6fe147550 (127.0.97.2:38251): Not found: Tablet not found: 9737c3ac517a4952ba55edf2b335e6a4
W20260620 02:55:42.903309 15449 leader_election.cc:341] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6494b6ffa1464d9cb2f168f618805049 (127.0.97.3:46245): Not found: Tablet not found: 9737c3ac517a4952ba55edf2b335e6a4
I20260620 02:55:42.903535 15449 leader_election.cc:302] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bebde3ed8c2b454495b46420279facf6; no voters: 0460c393c90f4abd993a05c6fe147550, 6494b6ffa1464d9cb2f168f618805049
I20260620 02:55:42.903865 15860 raft_consensus.cc:2726] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 02:55:43.336658 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:43.335934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.2:49867)
0620 02:55:43.336409 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:43.336498 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":221,"threads_started":1}
I20260620 02:55:43.792466 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:43.791565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.3:39697)
0620 02:55:43.792085 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:43.792244 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":259,"threads_started":1}
I20260620 02:55:43.880270 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:43.879517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.1:38793)
0620 02:55:43.879973 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:43.880060 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":248,"threads_started":1}
I20260620 02:55:44.338739 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:44.337696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.2:34129)
0620 02:55:44.338196 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:44.338539 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":274,"threads_started":1}
I20260620 02:55:44.794277 15865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:44.793521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.3:45013)
0620 02:55:44.794011 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:44.794098 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":236,"threads_started":1}
I20260620 02:55:44.881966 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:44.881227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.1:36387)
0620 02:55:44.881700 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:44.881789 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":255,"threads_started":1}
I20260620 02:55:45.340734 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:45.339877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.2:37381)
0620 02:55:45.340461 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:45.340554 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":311,"threads_started":1}
I20260620 02:55:45.796154 15868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:45.795344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.3:53803)
0620 02:55:45.795905 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:45.795992 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":287,"threads_started":1}
I20260620 02:55:45.883777 15869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:45.882986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.1:35973)
0620 02:55:45.883484 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:45.883582 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":247,"threads_started":1}
I20260620 02:55:46.342607 15870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:46.341739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.2:41129)
0620 02:55:46.342267 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:46.342357 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":284,"threads_started":1}
I20260620 02:55:46.798213 15871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:46.797353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.3:48091)
0620 02:55:46.797824 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:46.797917 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":270,"threads_started":1}
I20260620 02:55:46.885694 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:46.884762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.1:51437)
0620 02:55:46.885425 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:46.885522 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":404,"threads_started":1}
I20260620 02:55:47.344326 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:47.343582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.2:52573)
0620 02:55:47.344070 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:47.344152 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":233,"threads_started":1}
I20260620 02:55:47.800001 15874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:47.799258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.3:46313)
0620 02:55:47.799721 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:47.799804 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":244,"threads_started":1}
I20260620 02:55:47.887542 15875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:47.886784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.1:36663)
0620 02:55:47.887273 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:47.887360 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":247,"threads_started":1}
I20260620 02:55:48.001305 15876 raft_consensus.cc:482] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:55:48.001533 15876 raft_consensus.cc:504] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0460c393c90f4abd993a05c6fe147550" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38251 } } peers { permanent_uuid: "6494b6ffa1464d9cb2f168f618805049" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46245 } } peers { permanent_uuid: "bebde3ed8c2b454495b46420279facf6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43863 } }
I20260620 02:55:48.002604 15876 leader_election.cc:288] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0460c393c90f4abd993a05c6fe147550 (127.0.97.2:38251), 6494b6ffa1464d9cb2f168f618805049 (127.0.97.3:46245)
W20260620 02:55:48.003345 15449 leader_election.cc:341] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6494b6ffa1464d9cb2f168f618805049 (127.0.97.3:46245): Not found: Tablet not found: 9737c3ac517a4952ba55edf2b335e6a4
W20260620 02:55:48.003700 15448 leader_election.cc:341] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0460c393c90f4abd993a05c6fe147550 (127.0.97.2:38251): Not found: Tablet not found: 9737c3ac517a4952ba55edf2b335e6a4
I20260620 02:55:48.003813 15448 leader_election.cc:302] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bebde3ed8c2b454495b46420279facf6; no voters: 0460c393c90f4abd993a05c6fe147550, 6494b6ffa1464d9cb2f168f618805049
I20260620 02:55:48.004070 15876 raft_consensus.cc:2726] T 9737c3ac517a4952ba55edf2b335e6a4 P bebde3ed8c2b454495b46420279facf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 02:55:48.346046 15877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:48.345284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.2:44439)
0620 02:55:48.345763 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:48.345844 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":264,"threads_started":1}
I20260620 02:55:48.801831 15878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:48.800993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.3:42797)
0620 02:55:48.801529 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:48.801634 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":275,"threads_started":1}
I20260620 02:55:48.889608 15879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:48.888868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.1:47533)
0620 02:55:48.889346 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:48.889431 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":263,"threads_started":1}
I20260620 02:55:49.348001 15880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:49.347259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.2:41875)
0620 02:55:49.347720 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:49.347836 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":233,"threads_started":1}
I20260620 02:55:49.776227 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 15397
I20260620 02:55:49.798707 15881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:49.777400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36412 (local address 127.0.97.1:43863)
0620 02:55:49.777985 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:49.777996 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:49.778296 (+ 300us) server_negotiation.cc:407] Connection header received
0620 02:55:49.778769 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:49.778778 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:49.779126 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:49.779271 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:49.780765 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:49.781766 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:49.783303 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:49.783916 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:49.784034 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:49.792232 (+ 8198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:49.792247 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:49.792264 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:49.792303 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:49.795529 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:49.796378 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:49.796386 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:49.796389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:49.796537 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:49.796997 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:49.797001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:49.797003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:49.797420 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:49.797690 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:49.798044 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 02:55:49.798320 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":261,"threads_started":1}
I20260620 02:55:49.800192 15556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:55:49.803905 15883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:49.803038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.3:56697)
0620 02:55:49.803543 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:49.803666 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":269,"threads_started":1}
I20260620 02:55:49.893999 15884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:49.892890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45553 (local address 127.0.97.1:56989)
0620 02:55:49.893520 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:49.893687 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":284,"threads_started":1}
I20260620 02:55:49.894024 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 15397
I20260620 02:55:49.912110 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 15564
I20260620 02:55:49.927115 15885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:49.912960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57974 (local address 127.0.97.2:38251)
0620 02:55:49.913490 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:49.913494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:49.913511 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:55:49.913573 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:49.913577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:49.913637 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:49.913730 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:49.915329 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:49.916177 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:49.917511 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:49.917962 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:49.918069 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:49.920954 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:49.920960 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:49.920963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:49.920996 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:49.923841 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:49.924710 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:49.924721 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:49.924724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:49.924933 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:49.925430 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:49.925436 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:49.925439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:49.925740 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:49.926012 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:49.926476 (+ 464us) server_negotiation.cc:299] Negotiation successful
0620 02:55:49.926731 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":321,"threads_started":1}
I20260620 02:55:49.927958 15688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:55:50.013023 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 15564
I20260620 02:55:50.031282 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 15696
I20260620 02:55:50.045806 15887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:50.032223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52252 (local address 127.0.97.3:46245)
0620 02:55:50.032813 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:50.032818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:50.032834 (+ 16us) server_negotiation.cc:407] Connection header received
0620 02:55:50.032891 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:50.032894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:50.032967 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:50.033064 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:50.034541 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:50.035402 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:50.036714 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:50.037208 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:50.037318 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:50.040110 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:50.040115 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:50.040119 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:50.040177 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:50.043129 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:50.043787 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:50.043792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:50.043794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:50.043963 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:50.044434 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:50.044438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:50.044440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:50.044729 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:50.044968 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:50.045317 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 02:55:50.045482 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":313,"threads_started":1}
I20260620 02:55:50.046707 15819 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:55:50.131461 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 15696
2026-06-20T02:55:50Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24688 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:55:50 dist-test-slave-8p8c krb5kdc[15891](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:55:50 dist-test-slave-8p8c krb5kdc[15891](info): set up 2 sockets
Jun 20 02:55:50 dist-test-slave-8p8c krb5kdc[15891](info): commencing operation
krb5kdc: starting...
W20260620 02:55:51.251091 388 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:55:51 dist-test-slave-8p8c krb5kdc[15891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:55:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:55:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:55:51.507264 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34103
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:55:51.843343 15904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:55:51.843663 15904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:55:51.843832 15904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:55:51.851689 15904 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 02:55:51.851805 15904 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 02:55:51.851845 15904 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 02:55:51.851943 15904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:55:51.852034 15904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:55:51.852150 15904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:55:51.852214 15904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:55:51.852264 15904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:55:51.862473 15904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:55:51.863837 15904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:55:51.865247 15904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:55:51.878118 15910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:51.878144 15909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:51.878391 15912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:55:51.879040 15904 server_base.cc:1034] running on GCE node
I20260620 02:55:51.879803 15904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:55:51.881191 15904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:55:51.882519 15904 hybrid_clock.cc:648] HybridClock initialized: now 1781924151882400 us; error 86 us; skew 500 ppm
I20260620 02:55:51.882913 15904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:55:51 dist-test-slave-8p8c krb5kdc[15891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:55:51.886231 15904 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:55:51.888042 15904 webserver.cc:460] Webserver started at http://127.0.97.62:43851/ using document root <none> and password file <none>
I20260620 02:55:51.888799 15904 fs_manager.cc:359] Metadata directory not provided
I20260620 02:55:51.888935 15904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:55:51.889267 15904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:55:51.892397 15904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "0c70203eb93e498c94be7647fcb3ab45"
format_stamp: "Formatted at 2026-06-20 02:55:51 on dist-test-slave-8p8c"
server_key: "60105ab7a1ceaa2b659f095cbec104e4"
server_key_iv: "a98a41bdd6b08775fca4bb39eb4c8c85"
server_key_version: "encryptionkey@0"
I20260620 02:55:51.893208 15904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "0c70203eb93e498c94be7647fcb3ab45"
format_stamp: "Formatted at 2026-06-20 02:55:51 on dist-test-slave-8p8c"
server_key: "60105ab7a1ceaa2b659f095cbec104e4"
server_key_iv: "a98a41bdd6b08775fca4bb39eb4c8c85"
server_key_version: "encryptionkey@0"
I20260620 02:55:51.898983 15904 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 02:55:51.902786 15919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:51.904364 15904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:55:51.904587 15904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "0c70203eb93e498c94be7647fcb3ab45"
format_stamp: "Formatted at 2026-06-20 02:55:51 on dist-test-slave-8p8c"
server_key: "60105ab7a1ceaa2b659f095cbec104e4"
server_key_iv: "a98a41bdd6b08775fca4bb39eb4c8c85"
server_key_version: "encryptionkey@0"
I20260620 02:55:51.904829 15904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:55:51.952905 15904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:55:51.957939 15904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:55:51.958271 15904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:55:51.979838 15904 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34103
I20260620 02:55:51.979856 15970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34103 every 8 connection(s)
I20260620 02:55:51.981431 15904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:55:51.985486 15971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:55:51.991804 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 15904
I20260620 02:55:51.992060 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:55:51.992726 388 external_mini_cluster.cc:1428] Setting key 4a3a709d8be480014fb5237694eb2ece
I20260620 02:55:51.998574 15971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45: Bootstrap starting.
I20260620 02:55:52.001978 15971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45: Neither blocks nor log segments found. Creating new log.
I20260620 02:55:52.003170 15971 log.cc:824] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45: Log is configured to *not* fsync() on all Append() calls
I20260620 02:55:52.006371 15971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45: No bootstrap required, opened a new log
Jun 20 02:55:52 dist-test-slave-8p8c krb5kdc[15891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:55:52.015867 15971 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c70203eb93e498c94be7647fcb3ab45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34103 } }
I20260620 02:55:52.016340 15971 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:55:52.016464 15971 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c70203eb93e498c94be7647fcb3ab45, State: Initialized, Role: FOLLOWER
I20260620 02:55:52.017143 15971 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c70203eb93e498c94be7647fcb3ab45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34103 } }
I20260620 02:55:52.017423 15971 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:55:52.017565 15971 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:55:52.017771 15971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:55:52.019629 15974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:51.994929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60702 (local address 127.0.97.62:34103)
0620 02:55:51.995932 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:51.995945 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:51.995995 (+ 50us) server_negotiation.cc:407] Connection header received
0620 02:55:51.998015 (+ 2020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:51.998043 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:51.998761 (+ 718us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:51.999191 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:52.001407 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:52.002912 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:52.004723 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:52.005458 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:52.005664 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:52.010557 (+ 4893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:52.010565 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:52.010582 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:52.010631 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:52.014986 (+ 4355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:52.015790 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:52.015798 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:52.015804 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:52.015989 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:52.016454 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:52.016459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:52.016464 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:52.017178 (+ 714us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:52.017495 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:52.017953 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 02:55:52.018486 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":773,"thread_start_us":494,"threads_started":1}
I20260620 02:55:52.020412 15971 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c70203eb93e498c94be7647fcb3ab45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34103 } }
I20260620 02:55:52.020961 15971 leader_election.cc:302] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c70203eb93e498c94be7647fcb3ab45; no voters:
I20260620 02:55:52.021598 15971 leader_election.cc:288] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:55:52.021754 15976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:55:52.022476 15976 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [term 1 LEADER]: Becoming Leader. State: Replica: 0c70203eb93e498c94be7647fcb3ab45, State: Running, Role: LEADER
I20260620 02:55:52.023124 15976 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c70203eb93e498c94be7647fcb3ab45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34103 } }
I20260620 02:55:52.024698 15971 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:55:52.025981 15978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c70203eb93e498c94be7647fcb3ab45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c70203eb93e498c94be7647fcb3ab45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34103 } } }
I20260620 02:55:52.025981 15977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c70203eb93e498c94be7647fcb3ab45. Latest consensus state: current_term: 1 leader_uuid: "0c70203eb93e498c94be7647fcb3ab45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c70203eb93e498c94be7647fcb3ab45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34103 } } }
I20260620 02:55:52.026450 15977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [sys.catalog]: This master's current role is: LEADER
I20260620 02:55:52.026454 15978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [sys.catalog]: This master's current role is: LEADER
I20260620 02:55:52.028594 15981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:55:52.034287 15981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:55:52.043243 15981 catalog_manager.cc:1347] Generated new cluster ID: d9a6e47a066448ad9d89372750215479
I20260620 02:55:52.043373 15981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:55:52.064455 15981 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:55:52.066519 15981 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:55:52.075049 15981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45: Generated new TSK 0
I20260620 02:55:52.076148 15981 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:55:52.148206 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34103
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:55:52.491528 15997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:55:52.491829 15997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:55:52.492012 15997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:55:52.499486 15997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:55:52.499616 15997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:55:52.499732 15997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:55:52.515110 15997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:55:52.517220 15997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:55:52.519532 15997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:55:52.536855 16005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:52.537250 16003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:52.537774 16002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:55:52.538058 15997 server_base.cc:1034] running on GCE node
I20260620 02:55:52.539033 15997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:55:52.540481 15997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:55:52.541736 15997 hybrid_clock.cc:648] HybridClock initialized: now 1781924152541685 us; error 50 us; skew 500 ppm
I20260620 02:55:52.542205 15997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:55:52 dist-test-slave-8p8c krb5kdc[15891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:55:52.545771 15997 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:55:52.547701 15997 webserver.cc:460] Webserver started at http://127.0.97.1:45737/ using document root <none> and password file <none>
I20260620 02:55:52.548491 15997 fs_manager.cc:359] Metadata directory not provided
I20260620 02:55:52.548624 15997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:55:52.548982 15997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:55:52.552377 15997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/data/instance:
uuid: "5115eed9ba524de59a14e34a4d3f1593"
format_stamp: "Formatted at 2026-06-20 02:55:52 on dist-test-slave-8p8c"
server_key: "b3b4318230e90b0377a19228a2b3e49d"
server_key_iv: "a420dc8272a8c8bd1d616f5eb996ebb0"
server_key_version: "encryptionkey@0"
I20260620 02:55:52.553182 15997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5115eed9ba524de59a14e34a4d3f1593"
format_stamp: "Formatted at 2026-06-20 02:55:52 on dist-test-slave-8p8c"
server_key: "b3b4318230e90b0377a19228a2b3e49d"
server_key_iv: "a420dc8272a8c8bd1d616f5eb996ebb0"
server_key_version: "encryptionkey@0"
I20260620 02:55:52.559192 15997 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260620 02:55:52.562716 16012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:52.564069 15997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:55:52.564277 15997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/wal
uuid: "5115eed9ba524de59a14e34a4d3f1593"
format_stamp: "Formatted at 2026-06-20 02:55:52 on dist-test-slave-8p8c"
server_key: "b3b4318230e90b0377a19228a2b3e49d"
server_key_iv: "a420dc8272a8c8bd1d616f5eb996ebb0"
server_key_version: "encryptionkey@0"
I20260620 02:55:52.564488 15997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:55:52.577303 15997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:55:52.581290 15997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:55:52.581575 15997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:55:52.582605 15997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:55:52.583937 15997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:55:52.584048 15997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:52.584193 15997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:55:52.584264 15997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:52.618031 15997 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40417
I20260620 02:55:52.618081 16124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40417 every 8 connection(s)
I20260620 02:55:52.619382 15997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:55:52.625399 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 15997
I20260620 02:55:52.625617 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:55:52.626434 388 external_mini_cluster.cc:1428] Setting key 999e1ba81ac321295d8bb8028899ceb7
Jun 20 02:55:52 dist-test-slave-8p8c krb5kdc[15891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:55:52.644702 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:52.622041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41931 (local address 127.0.97.62:34103)
0620 02:55:52.622619 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:52.622623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:52.623788 (+ 1165us) server_negotiation.cc:407] Connection header received
0620 02:55:52.626707 (+ 2919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:52.626712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:52.626786 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:52.626888 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:52.628577 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:52.629427 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:52.632240 (+ 2813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:52.632667 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:52.632778 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:52.637309 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:52.637317 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:52.637326 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:52.637376 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:52.641050 (+ 3674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:52.641875 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:52.641883 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:52.641889 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:52.642048 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:52.642535 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:52.642541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:52.642546 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:52.642937 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:52.643104 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:52.643934 (+ 830us) server_negotiation.cc:299] Negotiation successful
0620 02:55:52.644220 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":284,"threads_started":1}
I20260620 02:55:52.645510 16128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:52.622662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.1:41931)
0620 02:55:52.623610 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:52.623657 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:52.626216 (+ 2559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:52.627199 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:52.627208 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:52.627758 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:55:52.628340 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:52.628357 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:52.629643 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:52.629652 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:52.632010 (+ 2358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:52.632050 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:52.632919 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:52.632928 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:52.633280 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:52.634148 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:55:52.634184 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:55:52.637003 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:55:52.641230 (+ 4227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:52.641239 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:52.641255 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:52.641725 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:52.642156 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:52.642159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:52.642161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:52.642354 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:52.643162 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:55:52.643169 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:55:52.643517 (+ 348us) client_negotiation.cc:769] Sending connection context
0620 02:55:52.643769 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 02:55:52.644209 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":296,"threads_started":1}
I20260620 02:55:52.647248 16125 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34103
I20260620 02:55:52.647603 16125 heartbeater.cc:464] Registering TS with master...
I20260620 02:55:52.648525 16125 heartbeater.cc:511] Master 127.0.97.62:34103 requested a full tablet report, sending...
I20260620 02:55:52.651041 15936 ts_manager.cc:194] Registered new tserver with Master: 5115eed9ba524de59a14e34a4d3f1593 (127.0.97.1:40417)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:55:52.652590 15936 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:55:52.685583 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34103
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:55:53.014622 16131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:55:53.014950 16131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:55:53.015129 16131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:55:53.022480 16131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:55:53.022643 16131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:55:53.022830 16131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:55:53.033226 16131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:55:53.034634 16131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:55:53.036015 16131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:55:53.050825 16137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:53.050858 16139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:53.051056 16136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:55:53.052328 16131 server_base.cc:1034] running on GCE node
I20260620 02:55:53.053140 16131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:55:53.054247 16131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:55:53.055477 16131 hybrid_clock.cc:648] HybridClock initialized: now 1781924153055422 us; error 37 us; skew 500 ppm
I20260620 02:55:53.055836 16131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:55:53 dist-test-slave-8p8c krb5kdc[15891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:55:53.059134 16131 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:55:53.060660 16131 webserver.cc:460] Webserver started at http://127.0.97.2:44397/ using document root <none> and password file <none>
I20260620 02:55:53.061367 16131 fs_manager.cc:359] Metadata directory not provided
I20260620 02:55:53.061517 16131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:55:53.061829 16131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:55:53.064790 16131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/data/instance:
uuid: "2f80cb4ae084422183967f584a2b52d0"
format_stamp: "Formatted at 2026-06-20 02:55:53 on dist-test-slave-8p8c"
server_key: "b75a451ad3e3e4f820f050cdd4a12b0d"
server_key_iv: "4039541c37ce32f4bef155f5d7ed0b79"
server_key_version: "encryptionkey@0"
I20260620 02:55:53.065565 16131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2f80cb4ae084422183967f584a2b52d0"
format_stamp: "Formatted at 2026-06-20 02:55:53 on dist-test-slave-8p8c"
server_key: "b75a451ad3e3e4f820f050cdd4a12b0d"
server_key_iv: "4039541c37ce32f4bef155f5d7ed0b79"
server_key_version: "encryptionkey@0"
I20260620 02:55:53.071007 16131 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 02:55:53.074851 16146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:53.076225 16131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:55:53.076400 16131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/wal
uuid: "2f80cb4ae084422183967f584a2b52d0"
format_stamp: "Formatted at 2026-06-20 02:55:53 on dist-test-slave-8p8c"
server_key: "b75a451ad3e3e4f820f050cdd4a12b0d"
server_key_iv: "4039541c37ce32f4bef155f5d7ed0b79"
server_key_version: "encryptionkey@0"
I20260620 02:55:53.076601 16131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:55:53.092273 16131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:55:53.096794 16131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:55:53.097100 16131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:55:53.098126 16131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:55:53.099519 16131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:55:53.099682 16131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:53.099995 16131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:55:53.100059 16131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:53.136543 16131 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46295
I20260620 02:55:53.136561 16258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46295 every 8 connection(s)
I20260620 02:55:53.137842 16131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:55:53.139338 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 16131
I20260620 02:55:53.139545 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:55:53.140079 388 external_mini_cluster.cc:1428] Setting key 9d706f30f9c9ced20ada7ae7fe8b0127
Jun 20 02:55:53 dist-test-slave-8p8c krb5kdc[15891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:55:53.160102 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:53.140886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42899 (local address 127.0.97.62:34103)
0620 02:55:53.141085 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:53.141090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:53.142537 (+ 1447us) server_negotiation.cc:407] Connection header received
0620 02:55:53.144361 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:53.144366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:53.144454 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:53.144583 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:53.146142 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.146959 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.148529 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.149321 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.149589 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:53.153951 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:53.153957 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:53.153960 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:53.153994 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:53.156761 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:53.157544 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:53.157556 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:53.157559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:53.157684 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:53.158134 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:53.158141 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:53.158144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:53.158498 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:53.158654 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:53.159513 (+ 859us) server_negotiation.cc:299] Negotiation successful
0620 02:55:53.159755 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 02:55:53.161185 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:53.141471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.2:42899)
0620 02:55:53.142332 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:53.142421 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:53.144096 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:53.144827 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:53.144835 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:53.145379 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:55:53.145962 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:53.145991 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.147124 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.147135 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:53.148370 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:53.148382 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.149556 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.149561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:53.149784 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:53.150577 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:55:53.150617 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:55:53.153690 (+ 3073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:55:53.156935 (+ 3245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:53.156944 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:53.156963 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:53.157407 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:53.157799 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:53.157804 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:53.157808 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:53.158020 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:53.158627 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:55:53.158636 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:55:53.159032 (+ 396us) client_negotiation.cc:769] Sending connection context
0620 02:55:53.159372 (+ 340us) client_negotiation.cc:240] Negotiation successful
0620 02:55:53.159965 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":346,"threads_started":1}
I20260620 02:55:53.162922 16259 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34103
I20260620 02:55:53.163301 16259 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:55:53.164347 16259 heartbeater.cc:511] Master 127.0.97.62:34103 requested a full tablet report, sending...
I20260620 02:55:53.166041 15936 ts_manager.cc:194] Registered new tserver with Master: 2f80cb4ae084422183967f584a2b52d0 (127.0.97.2:46295)
I20260620 02:55:53.167392 15936 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42899
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:55:53.198968 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34103
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:55:53.530582 16264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:55:53.530892 16264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:55:53.531060 16264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:55:53.538311 16264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:55:53.538483 16264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:55:53.538652 16264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:55:53.548276 16264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:55:53.549638 16264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:55:53.551118 16264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:55:53.564451 16269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:53.564790 16270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:55:53.564837 16272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:55:53.565526 16264 server_base.cc:1034] running on GCE node
I20260620 02:55:53.566136 16264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:55:53.567183 16264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:55:53.568389 16264 hybrid_clock.cc:648] HybridClock initialized: now 1781924153568350 us; error 36 us; skew 500 ppm
I20260620 02:55:53.568745 16264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:55:53 dist-test-slave-8p8c krb5kdc[15891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:55:53.571966 16264 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:55:53.573498 16264 webserver.cc:460] Webserver started at http://127.0.97.3:39989/ using document root <none> and password file <none>
I20260620 02:55:53.574134 16264 fs_manager.cc:359] Metadata directory not provided
I20260620 02:55:53.574234 16264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:55:53.574559 16264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:55:53.577366 16264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/data/instance:
uuid: "85367245efeb4a08b1599f5ffcced30c"
format_stamp: "Formatted at 2026-06-20 02:55:53 on dist-test-slave-8p8c"
server_key: "9717a6c83bf6296baa55c03fe57e9bd1"
server_key_iv: "676b9c17c40aa29b562bf1f6073e92b4"
server_key_version: "encryptionkey@0"
I20260620 02:55:53.578058 16264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/wal/instance:
uuid: "85367245efeb4a08b1599f5ffcced30c"
format_stamp: "Formatted at 2026-06-20 02:55:53 on dist-test-slave-8p8c"
server_key: "9717a6c83bf6296baa55c03fe57e9bd1"
server_key_iv: "676b9c17c40aa29b562bf1f6073e92b4"
server_key_version: "encryptionkey@0"
I20260620 02:55:53.583177 16264 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 02:55:53.586760 16279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:53.588148 16264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:55:53.588327 16264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/wal
uuid: "85367245efeb4a08b1599f5ffcced30c"
format_stamp: "Formatted at 2026-06-20 02:55:53 on dist-test-slave-8p8c"
server_key: "9717a6c83bf6296baa55c03fe57e9bd1"
server_key_iv: "676b9c17c40aa29b562bf1f6073e92b4"
server_key_version: "encryptionkey@0"
I20260620 02:55:53.588517 16264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:55:53.606319 16264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:55:53.611034 16264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:55:53.611327 16264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:55:53.612345 16264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:55:53.613653 16264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:55:53.613729 16264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:53.613862 16264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:55:53.613912 16264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:55:53.648283 16264 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45263
I20260620 02:55:53.648327 16391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45263 every 8 connection(s)
I20260620 02:55:53.649550 16264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:55:53.652503 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 16264
I20260620 02:55:53.652702 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:55:53.653172 388 external_mini_cluster.cc:1428] Setting key bd3d8ce211dc0341807fea15cf54b1fb
I20260620 02:55:53.656077 16125 heartbeater.cc:503] Master 127.0.97.62:34103 was elected leader, sending a full tablet report...
Jun 20 02:55:53 dist-test-slave-8p8c krb5kdc[15891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:55:53.670392 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:53.652159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53767 (local address 127.0.97.62:34103)
0620 02:55:53.652402 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:53.652409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:53.653694 (+ 1285us) server_negotiation.cc:407] Connection header received
0620 02:55:53.655617 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:53.655623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:53.655708 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:53.655824 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:53.657618 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.658829 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.660441 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.660848 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.660996 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:53.664472 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:53.664479 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:53.664484 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:53.664525 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:53.667222 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:53.667980 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:53.667987 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:53.667992 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:53.668174 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:53.668604 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:53.668609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:53.668612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:53.668893 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:53.669052 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:53.669791 (+ 739us) server_negotiation.cc:299] Negotiation successful
0620 02:55:53.670022 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 02:55:53.671303 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:53.652656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.3:53767)
0620 02:55:53.653529 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:53.653581 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:53.655347 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:53.656068 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:53.656076 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:53.656816 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:55:53.657449 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:53.657483 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.659026 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.659030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:53.660296 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:53.660308 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.661074 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.661084 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:53.661333 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:53.662040 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:55:53.662067 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:55:53.664269 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:55:53.667357 (+ 3088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:53.667367 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:53.667383 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:53.667865 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:53.668280 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:55:53.668286 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:55:53.668289 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:55:53.668468 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:55:53.669014 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:55:53.669022 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:55:53.669427 (+ 405us) client_negotiation.cc:769] Sending connection context
0620 02:55:53.669695 (+ 268us) client_negotiation.cc:240] Negotiation successful
0620 02:55:53.670197 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":358,"threads_started":1}
I20260620 02:55:53.672605 16392 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34103
I20260620 02:55:53.673041 16392 heartbeater.cc:464] Registering TS with master...
I20260620 02:55:53.673760 16392 heartbeater.cc:511] Master 127.0.97.62:34103 requested a full tablet report, sending...
I20260620 02:55:53.675244 15936 ts_manager.cc:194] Registered new tserver with Master: 85367245efeb4a08b1599f5ffcced30c (127.0.97.3:45263)
I20260620 02:55:53.676076 15936 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53767
I20260620 02:55:53.680696 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:55:53.698310 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:53.685336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51146 (local address 127.0.97.62:34103)
0620 02:55:53.685579 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:53.685587 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:53.685921 (+ 334us) server_negotiation.cc:407] Connection header received
0620 02:55:53.686133 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:53.686138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:53.686206 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:53.686307 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:53.687759 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.688480 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.689851 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.690261 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.690423 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:53.693006 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:53.693011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:53.693014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:53.693047 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:53.695731 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:53.696369 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:53.696377 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:53.696380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:53.696507 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:53.696881 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:53.696886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:53.696889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:53.697193 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:53.697457 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:53.697764 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 02:55:53.697998 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 02:55:53.702721 15936 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:55:53.709728 15936 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:55:53.735644 16404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:53.723544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40417 (local address 127.0.0.1:41818)
0620 02:55:53.726074 (+ 2530us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:53.726093 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:53.726254 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:53.727461 (+ 1207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:53.727473 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:53.727523 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:55:53.728094 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:53.728110 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.730177 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.730182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:53.732508 (+ 2326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:53.732520 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.734087 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.734095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:53.734622 (+ 527us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:53.734693 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 02:55:53.734842 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 02:55:53.735049 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2387,"mutex_wait_us":1,"thread_start_us":787,"threads_started":1}
I20260620 02:55:53.735976 16407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:53.724155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41818 (local address 127.0.97.1:40417)
0620 02:55:53.725786 (+ 1631us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:53.725800 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:53.726455 (+ 655us) server_negotiation.cc:407] Connection header received
0620 02:55:53.726568 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:53.726576 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:53.726930 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:53.727151 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:55:53.728309 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.730025 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.732668 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.733880 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.734024 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:53.734262 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:53.735441 (+ 1179us) server_negotiation.cc:299] Negotiation successful
0620 02:55:53.735642 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1380,"thread_start_us":1120,"threads_started":1}
I20260620 02:55:53.736557 16406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:53.722789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45263 (local address 127.0.0.1:39048)
0620 02:55:53.724547 (+ 1758us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:53.724601 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:53.724848 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:53.728597 (+ 3749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:53.728605 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:53.728643 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:55:53.729243 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:53.729259 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.731465 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.731474 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:53.733692 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:53.733709 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.735669 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.735678 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:53.735863 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:53.735898 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 02:55:53.735972 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 02:55:53.736115 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1359,"mutex_wait_us":25,"thread_start_us":3002,"threads_started":1}
I20260620 02:55:53.737131 16405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:53.723131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39048 (local address 127.0.97.3:45263)
0620 02:55:53.727915 (+ 4784us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:53.727928 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:53.727966 (+ 38us) server_negotiation.cc:407] Connection header received
0620 02:55:53.728077 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:53.728085 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:53.728333 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:53.728571 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:55:53.729503 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.731307 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.733864 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.735387 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.736233 (+ 846us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:53.736483 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:53.736573 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 02:55:53.736752 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4610,"thread_start_us":4451,"threads_started":1}
I20260620 02:55:53.738595 16409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:53.727190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46295 (local address 127.0.0.1:52874)
0620 02:55:53.729835 (+ 2645us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:53.729849 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:53.729983 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:53.730941 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:53.730947 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:53.730977 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:55:53.731387 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:53.731400 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.733381 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.733385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:53.735465 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:53.735479 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.737816 (+ 2337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.737823 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:53.737994 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:53.738027 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 02:55:53.738118 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 02:55:53.738250 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2534,"thread_start_us":2365,"threads_started":1}
I20260620 02:55:53.738701 16408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:53.726456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52874 (local address 127.0.97.2:46295)
0620 02:55:53.728252 (+ 1796us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:53.728264 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:53.730122 (+ 1858us) server_negotiation.cc:407] Connection header received
0620 02:55:53.730238 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:53.730245 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:53.730513 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:53.730714 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:55:53.731565 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.733219 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.735652 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.737613 (+ 1961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.737768 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:53.737961 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:53.738190 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 02:55:53.738351 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1645,"thread_start_us":1527,"threads_started":1}
I20260620 02:55:53.740443 16060 tablet_service.cc:1467] Processing CreateTablet for tablet cef70bfe6d8d4d1881c595270a03568e (DEFAULT_TABLE table=test-table [id=98e719cddd7843e0ba6392f392fdde78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:55:53.740971 16327 tablet_service.cc:1467] Processing CreateTablet for tablet cef70bfe6d8d4d1881c595270a03568e (DEFAULT_TABLE table=test-table [id=98e719cddd7843e0ba6392f392fdde78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:55:53.742017 16060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef70bfe6d8d4d1881c595270a03568e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:55:53.742406 16327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef70bfe6d8d4d1881c595270a03568e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:55:53.742094 16194 tablet_service.cc:1467] Processing CreateTablet for tablet cef70bfe6d8d4d1881c595270a03568e (DEFAULT_TABLE table=test-table [id=98e719cddd7843e0ba6392f392fdde78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:55:53.743213 16194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef70bfe6d8d4d1881c595270a03568e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:55:53.754571 16410 tablet_bootstrap.cc:492] T cef70bfe6d8d4d1881c595270a03568e P 2f80cb4ae084422183967f584a2b52d0: Bootstrap starting.
I20260620 02:55:53.756713 16411 tablet_bootstrap.cc:492] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593: Bootstrap starting.
I20260620 02:55:53.758569 16410 tablet_bootstrap.cc:654] T cef70bfe6d8d4d1881c595270a03568e P 2f80cb4ae084422183967f584a2b52d0: Neither blocks nor log segments found. Creating new log.
I20260620 02:55:53.759912 16410 log.cc:824] T cef70bfe6d8d4d1881c595270a03568e P 2f80cb4ae084422183967f584a2b52d0: Log is configured to *not* fsync() on all Append() calls
I20260620 02:55:53.760408 16411 tablet_bootstrap.cc:654] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593: Neither blocks nor log segments found. Creating new log.
I20260620 02:55:53.760907 16412 tablet_bootstrap.cc:492] T cef70bfe6d8d4d1881c595270a03568e P 85367245efeb4a08b1599f5ffcced30c: Bootstrap starting.
I20260620 02:55:53.761936 16411 log.cc:824] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593: Log is configured to *not* fsync() on all Append() calls
I20260620 02:55:53.765018 16412 tablet_bootstrap.cc:654] T cef70bfe6d8d4d1881c595270a03568e P 85367245efeb4a08b1599f5ffcced30c: Neither blocks nor log segments found. Creating new log.
I20260620 02:55:53.765514 16411 tablet_bootstrap.cc:492] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593: No bootstrap required, opened a new log
I20260620 02:55:53.765799 16411 ts_tablet_manager.cc:1397] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 02:55:53.766278 16412 log.cc:824] T cef70bfe6d8d4d1881c595270a03568e P 85367245efeb4a08b1599f5ffcced30c: Log is configured to *not* fsync() on all Append() calls
I20260620 02:55:53.766453 16410 tablet_bootstrap.cc:492] T cef70bfe6d8d4d1881c595270a03568e P 2f80cb4ae084422183967f584a2b52d0: No bootstrap required, opened a new log
I20260620 02:55:53.766738 16410 ts_tablet_manager.cc:1397] T cef70bfe6d8d4d1881c595270a03568e P 2f80cb4ae084422183967f584a2b52d0: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.003s
I20260620 02:55:53.769510 16412 tablet_bootstrap.cc:492] T cef70bfe6d8d4d1881c595270a03568e P 85367245efeb4a08b1599f5ffcced30c: No bootstrap required, opened a new log
I20260620 02:55:53.769768 16412 ts_tablet_manager.cc:1397] T cef70bfe6d8d4d1881c595270a03568e P 85367245efeb4a08b1599f5ffcced30c: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260620 02:55:53.774528 16411 raft_consensus.cc:348] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85367245efeb4a08b1599f5ffcced30c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45263 } } peers { permanent_uuid: "2f80cb4ae084422183967f584a2b52d0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46295 } } peers { permanent_uuid: "5115eed9ba524de59a14e34a4d3f1593" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40417 } }
I20260620 02:55:53.774937 16411 raft_consensus.cc:374] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:55:53.775038 16411 raft_consensus.cc:729] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5115eed9ba524de59a14e34a4d3f1593, State: Initialized, Role: FOLLOWER
I20260620 02:55:53.775669 16411 consensus_queue.cc:260] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85367245efeb4a08b1599f5ffcced30c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45263 } } peers { permanent_uuid: "2f80cb4ae084422183967f584a2b52d0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46295 } } peers { permanent_uuid: "5115eed9ba524de59a14e34a4d3f1593" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40417 } }
I20260620 02:55:53.777251 16410 raft_consensus.cc:348] T cef70bfe6d8d4d1881c595270a03568e P 2f80cb4ae084422183967f584a2b52d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85367245efeb4a08b1599f5ffcced30c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45263 } } peers { permanent_uuid: "2f80cb4ae084422183967f584a2b52d0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46295 } } peers { permanent_uuid: "5115eed9ba524de59a14e34a4d3f1593" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40417 } }
I20260620 02:55:53.777571 16411 ts_tablet_manager.cc:1428] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260620 02:55:53.777678 16410 raft_consensus.cc:374] T cef70bfe6d8d4d1881c595270a03568e P 2f80cb4ae084422183967f584a2b52d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:55:53.777765 16410 raft_consensus.cc:729] T cef70bfe6d8d4d1881c595270a03568e P 2f80cb4ae084422183967f584a2b52d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f80cb4ae084422183967f584a2b52d0, State: Initialized, Role: FOLLOWER
I20260620 02:55:53.778358 16410 consensus_queue.cc:260] T cef70bfe6d8d4d1881c595270a03568e P 2f80cb4ae084422183967f584a2b52d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85367245efeb4a08b1599f5ffcced30c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45263 } } peers { permanent_uuid: "2f80cb4ae084422183967f584a2b52d0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46295 } } peers { permanent_uuid: "5115eed9ba524de59a14e34a4d3f1593" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40417 } }
I20260620 02:55:53.779548 16259 heartbeater.cc:503] Master 127.0.97.62:34103 was elected leader, sending a full tablet report...
I20260620 02:55:53.780136 16410 ts_tablet_manager.cc:1428] T cef70bfe6d8d4d1881c595270a03568e P 2f80cb4ae084422183967f584a2b52d0: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260620 02:55:53.780579 16412 raft_consensus.cc:348] T cef70bfe6d8d4d1881c595270a03568e P 85367245efeb4a08b1599f5ffcced30c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85367245efeb4a08b1599f5ffcced30c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45263 } } peers { permanent_uuid: "2f80cb4ae084422183967f584a2b52d0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46295 } } peers { permanent_uuid: "5115eed9ba524de59a14e34a4d3f1593" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40417 } }
I20260620 02:55:53.781072 16412 raft_consensus.cc:374] T cef70bfe6d8d4d1881c595270a03568e P 85367245efeb4a08b1599f5ffcced30c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:55:53.781183 16412 raft_consensus.cc:729] T cef70bfe6d8d4d1881c595270a03568e P 85367245efeb4a08b1599f5ffcced30c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85367245efeb4a08b1599f5ffcced30c, State: Initialized, Role: FOLLOWER
I20260620 02:55:53.781824 16412 consensus_queue.cc:260] T cef70bfe6d8d4d1881c595270a03568e P 85367245efeb4a08b1599f5ffcced30c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85367245efeb4a08b1599f5ffcced30c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45263 } } peers { permanent_uuid: "2f80cb4ae084422183967f584a2b52d0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46295 } } peers { permanent_uuid: "5115eed9ba524de59a14e34a4d3f1593" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40417 } }
I20260620 02:55:53.783100 16392 heartbeater.cc:503] Master 127.0.97.62:34103 was elected leader, sending a full tablet report...
I20260620 02:55:53.784912 16412 ts_tablet_manager.cc:1428] T cef70bfe6d8d4d1881c595270a03568e P 85367245efeb4a08b1599f5ffcced30c: Time spent starting tablet: real 0.015s user 0.013s sys 0.002s
I20260620 02:55:53.818125 16416 raft_consensus.cc:482] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:55:53.818524 16416 raft_consensus.cc:504] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85367245efeb4a08b1599f5ffcced30c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45263 } } peers { permanent_uuid: "2f80cb4ae084422183967f584a2b52d0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46295 } } peers { permanent_uuid: "5115eed9ba524de59a14e34a4d3f1593" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40417 } }
I20260620 02:55:53.820236 16416 leader_election.cc:288] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85367245efeb4a08b1599f5ffcced30c (127.0.97.3:45263), 2f80cb4ae084422183967f584a2b52d0 (127.0.97.2:46295)
I20260620 02:55:53.828224 16405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:53.820920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45727 (local address 127.0.97.3:45263)
0620 02:55:53.821132 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:53.821137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:53.821661 (+ 524us) server_negotiation.cc:407] Connection header received
0620 02:55:53.821902 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:53.821911 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:53.822016 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:53.822117 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:55:53.822958 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.824021 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.825917 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.827315 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.827418 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:53.827510 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:53.827912 (+ 402us) server_negotiation.cc:299] Negotiation successful
0620 02:55:53.828016 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 02:55:53.828413 16419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:53.820735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45263 (local address 127.0.97.1:45727)
0620 02:55:53.821531 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:53.821552 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:53.821701 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:53.822392 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:53.822400 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:53.822426 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:55:53.822808 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:53.822819 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.824139 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.824147 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:53.825713 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:53.825725 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.827523 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.827530 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:53.827717 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:53.827753 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 02:55:53.827864 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 02:55:53.828006 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":3,"thread_start_us":477,"threads_started":1}
I20260620 02:55:53.829648 16420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:53.821686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46295 (local address 127.0.97.1:55221)
0620 02:55:53.822683 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:53.822701 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:55:53.822843 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:55:53.823256 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:55:53.823261 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:55:53.823288 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:55:53.823623 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:53.823634 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.824821 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.824829 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:53.827043 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:55:53.827060 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.829038 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.829043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:55:53.829165 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:53.829187 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 02:55:53.829266 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 02:55:53.829351 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":859,"thread_start_us":609,"threads_started":1}
I20260620 02:55:53.829562 16347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef70bfe6d8d4d1881c595270a03568e" candidate_uuid: "5115eed9ba524de59a14e34a4d3f1593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85367245efeb4a08b1599f5ffcced30c" is_pre_election: true
I20260620 02:55:53.829763 16408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:53.821892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55221 (local address 127.0.97.2:46295)
0620 02:55:53.822135 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:53.822141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:53.822815 (+ 674us) server_negotiation.cc:407] Connection header received
0620 02:55:53.823043 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:53.823047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:53.823108 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:53.823218 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:55:53.823815 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.824621 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.827230 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.828454 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.828619 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:53.828774 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:53.829340 (+ 566us) server_negotiation.cc:299] Negotiation successful
0620 02:55:53.829484 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260620 02:55:53.830226 16347 raft_consensus.cc:2454] T cef70bfe6d8d4d1881c595270a03568e P 85367245efeb4a08b1599f5ffcced30c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5115eed9ba524de59a14e34a4d3f1593 in term 0.
I20260620 02:55:53.830915 16214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef70bfe6d8d4d1881c595270a03568e" candidate_uuid: "5115eed9ba524de59a14e34a4d3f1593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f80cb4ae084422183967f584a2b52d0" is_pre_election: true
I20260620 02:55:53.831035 16014 leader_election.cc:302] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5115eed9ba524de59a14e34a4d3f1593, 85367245efeb4a08b1599f5ffcced30c; no voters:
I20260620 02:55:53.831511 16416 raft_consensus.cc:2781] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:55:53.831491 16214 raft_consensus.cc:2454] T cef70bfe6d8d4d1881c595270a03568e P 2f80cb4ae084422183967f584a2b52d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5115eed9ba524de59a14e34a4d3f1593 in term 0.
I20260620 02:55:53.831679 16416 raft_consensus.cc:482] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:55:53.831789 16416 raft_consensus.cc:3037] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:55:53.834014 16416 raft_consensus.cc:504] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85367245efeb4a08b1599f5ffcced30c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45263 } } peers { permanent_uuid: "2f80cb4ae084422183967f584a2b52d0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46295 } } peers { permanent_uuid: "5115eed9ba524de59a14e34a4d3f1593" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40417 } }
I20260620 02:55:53.834717 16416 leader_election.cc:288] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593 [CANDIDATE]: Term 1 election: Requested vote from peers 85367245efeb4a08b1599f5ffcced30c (127.0.97.3:45263), 2f80cb4ae084422183967f584a2b52d0 (127.0.97.2:46295)
I20260620 02:55:53.835245 16347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef70bfe6d8d4d1881c595270a03568e" candidate_uuid: "5115eed9ba524de59a14e34a4d3f1593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85367245efeb4a08b1599f5ffcced30c"
I20260620 02:55:53.835351 16214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef70bfe6d8d4d1881c595270a03568e" candidate_uuid: "5115eed9ba524de59a14e34a4d3f1593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f80cb4ae084422183967f584a2b52d0"
I20260620 02:55:53.835453 16347 raft_consensus.cc:3037] T cef70bfe6d8d4d1881c595270a03568e P 85367245efeb4a08b1599f5ffcced30c [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:55:53.835595 16214 raft_consensus.cc:3037] T cef70bfe6d8d4d1881c595270a03568e P 2f80cb4ae084422183967f584a2b52d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:55:53.837692 16347 raft_consensus.cc:2454] T cef70bfe6d8d4d1881c595270a03568e P 85367245efeb4a08b1599f5ffcced30c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5115eed9ba524de59a14e34a4d3f1593 in term 1.
I20260620 02:55:53.837857 16214 raft_consensus.cc:2454] T cef70bfe6d8d4d1881c595270a03568e P 2f80cb4ae084422183967f584a2b52d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5115eed9ba524de59a14e34a4d3f1593 in term 1.
I20260620 02:55:53.838162 16014 leader_election.cc:302] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5115eed9ba524de59a14e34a4d3f1593, 85367245efeb4a08b1599f5ffcced30c; no voters:
I20260620 02:55:53.838476 16416 raft_consensus.cc:2781] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:55:53.838958 16416 raft_consensus.cc:686] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593 [term 1 LEADER]: Becoming Leader. State: Replica: 5115eed9ba524de59a14e34a4d3f1593, State: Running, Role: LEADER
I20260620 02:55:53.839424 16416 consensus_queue.cc:237] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85367245efeb4a08b1599f5ffcced30c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45263 } } peers { permanent_uuid: "2f80cb4ae084422183967f584a2b52d0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46295 } } peers { permanent_uuid: "5115eed9ba524de59a14e34a4d3f1593" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40417 } }
I20260620 02:55:53.844501 15935 catalog_manager.cc:5495] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5115eed9ba524de59a14e34a4d3f1593 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "5115eed9ba524de59a14e34a4d3f1593" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85367245efeb4a08b1599f5ffcced30c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f80cb4ae084422183967f584a2b52d0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5115eed9ba524de59a14e34a4d3f1593" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40417 } health_report { overall_health: HEALTHY } } }
W20260620 02:55:53.871802 16126 tablet.cc:2369] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:55:53.890084 16407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:53.884640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41826 (local address 127.0.97.1:40417)
0620 02:55:53.884829 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:53.884833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:53.884904 (+ 71us) server_negotiation.cc:407] Connection header received
0620 02:55:53.885072 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:53.885075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:53.885156 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:53.885285 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:55:53.885625 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.886563 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.887933 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.888380 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.888488 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:53.889042 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:55:53.889557 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:55:53.889682 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:53.889772 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 02:55:53.889842 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 02:55:53.891134 16260 tablet.cc:2369] T cef70bfe6d8d4d1881c595270a03568e P 2f80cb4ae084422183967f584a2b52d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:55:53.898603 16347 raft_consensus.cc:1261] T cef70bfe6d8d4d1881c595270a03568e P 85367245efeb4a08b1599f5ffcced30c [term 1 FOLLOWER]: Refusing update from remote peer 5115eed9ba524de59a14e34a4d3f1593: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:55:53.898638 16214 raft_consensus.cc:1261] T cef70bfe6d8d4d1881c595270a03568e P 2f80cb4ae084422183967f584a2b52d0 [term 1 FOLLOWER]: Refusing update from remote peer 5115eed9ba524de59a14e34a4d3f1593: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:55:53.899601 16421 consensus_queue.cc:1035] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f80cb4ae084422183967f584a2b52d0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:55:53.900084 16416 consensus_queue.cc:1035] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85367245efeb4a08b1599f5ffcced30c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 02:55:53.901209 16393 tablet.cc:2369] T cef70bfe6d8d4d1881c595270a03568e P 85367245efeb4a08b1599f5ffcced30c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:55:53.927114 15935 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:55:53.927403 15935 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:55:53.931422 15935 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45: Sending DeleteTablet for 3 replicas of tablet cef70bfe6d8d4d1881c595270a03568e
I20260620 02:55:53.932727 16194 tablet_service.cc:1514] Processing DeleteTablet for tablet cef70bfe6d8d4d1881c595270a03568e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:55:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52874
I20260620 02:55:53.932737 16060 tablet_service.cc:1514] Processing DeleteTablet for tablet cef70bfe6d8d4d1881c595270a03568e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:55:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41818
I20260620 02:55:53.932765 16327 tablet_service.cc:1514] Processing DeleteTablet for tablet cef70bfe6d8d4d1881c595270a03568e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:55:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39048
I20260620 02:55:53.934082 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 15997
I20260620 02:55:53.960425 16434 tablet_replica.cc:330] T cef70bfe6d8d4d1881c595270a03568e P 85367245efeb4a08b1599f5ffcced30c: stopping tablet replica
I20260620 02:55:53.964637 16434 raft_consensus.cc:2229] T cef70bfe6d8d4d1881c595270a03568e P 85367245efeb4a08b1599f5ffcced30c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:55:53.965461 16434 raft_consensus.cc:2258] T cef70bfe6d8d4d1881c595270a03568e P 85367245efeb4a08b1599f5ffcced30c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:55:53.969993 16434 ts_tablet_manager.cc:1905] T cef70bfe6d8d4d1881c595270a03568e P 85367245efeb4a08b1599f5ffcced30c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:55:53.971997 16433 tablet_replica.cc:330] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593: stopping tablet replica
I20260620 02:55:53.972661 16435 tablet_replica.cc:330] T cef70bfe6d8d4d1881c595270a03568e P 2f80cb4ae084422183967f584a2b52d0: stopping tablet replica
I20260620 02:55:53.973136 16435 raft_consensus.cc:2229] T cef70bfe6d8d4d1881c595270a03568e P 2f80cb4ae084422183967f584a2b52d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:55:53.973724 16435 raft_consensus.cc:2258] T cef70bfe6d8d4d1881c595270a03568e P 2f80cb4ae084422183967f584a2b52d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:55:53.974504 16433 raft_consensus.cc:2229] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:55:53.975224 16433 raft_consensus.cc:2258] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:55:53.977025 16434 ts_tablet_manager.cc:1918] T cef70bfe6d8d4d1881c595270a03568e P 85367245efeb4a08b1599f5ffcced30c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:55:53.977187 16434 log.cc:1196] T cef70bfe6d8d4d1881c595270a03568e P 85367245efeb4a08b1599f5ffcced30c: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/wal/wals/cef70bfe6d8d4d1881c595270a03568e
Jun 20 02:55:53 dist-test-slave-8p8c krb5kdc[15891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:55:53.977676 16434 ts_tablet_manager.cc:1939] T cef70bfe6d8d4d1881c595270a03568e P 85367245efeb4a08b1599f5ffcced30c: Deleting consensus metadata
I20260620 02:55:53.978624 16435 ts_tablet_manager.cc:1905] T cef70bfe6d8d4d1881c595270a03568e P 2f80cb4ae084422183967f584a2b52d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:55:53.979700 15921 catalog_manager.cc:4841] TS 85367245efeb4a08b1599f5ffcced30c (127.0.97.3:45263): tablet cef70bfe6d8d4d1881c595270a03568e (table test-table [id=98e719cddd7843e0ba6392f392fdde78]) successfully deleted
I20260620 02:55:53.985275 16433 ts_tablet_manager.cc:1905] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:55:53.986027 16435 ts_tablet_manager.cc:1918] T cef70bfe6d8d4d1881c595270a03568e P 2f80cb4ae084422183967f584a2b52d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:55:53.986181 16435 log.cc:1196] T cef70bfe6d8d4d1881c595270a03568e P 2f80cb4ae084422183967f584a2b52d0: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/wal/wals/cef70bfe6d8d4d1881c595270a03568e
I20260620 02:55:53.987128 16435 ts_tablet_manager.cc:1939] T cef70bfe6d8d4d1881c595270a03568e P 2f80cb4ae084422183967f584a2b52d0: Deleting consensus metadata
I20260620 02:55:53.989327 15921 catalog_manager.cc:4841] TS 2f80cb4ae084422183967f584a2b52d0 (127.0.97.2:46295): tablet cef70bfe6d8d4d1881c595270a03568e (table test-table [id=98e719cddd7843e0ba6392f392fdde78]) successfully deleted
I20260620 02:55:53.993957 16407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:53.935346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41842 (local address 127.0.97.1:40417)
0620 02:55:53.935607 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:53.935617 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:53.957151 (+ 21534us) server_negotiation.cc:407] Connection header received
0620 02:55:53.957673 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:53.957686 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:53.957777 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:53.957857 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:53.960667 (+ 2810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.962002 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.967428 (+ 5426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:53.968177 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:53.968317 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:53.985607 (+ 17290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:53.985621 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:53.985640 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:53.985688 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:53.989945 (+ 4257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:53.991017 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:53.991028 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:53.991031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:53.991225 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:53.991763 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:53.991769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:53.991772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:53.992255 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:53.992581 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:53.993264 (+ 683us) server_negotiation.cc:299] Negotiation successful
0620 02:55:53.993550 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 02:55:53.995613 16120 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:55:54.097944 16433 ts_tablet_manager.cc:1918] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:55:54.098268 16433 log.cc:1196] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/wal/wals/cef70bfe6d8d4d1881c595270a03568e
I20260620 02:55:54.098733 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 15997
I20260620 02:55:54.117951 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 16131
W20260620 02:55:54.118083 15923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:40417 (error 108)
I20260620 02:55:54.120007 16409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:54.119596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40417 (local address 127.0.0.1:41844)
0620 02:55:54.119766 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:54.119873 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 02:55:54.120220 15923 catalog_manager.cc:4584] TS 5115eed9ba524de59a14e34a4d3f1593 (127.0.97.1:40417): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cef70bfe6d8d4d1881c595270a03568e: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40417: connect: Connection refused (error 111)
Jun 20 02:55:54 dist-test-slave-8p8c krb5kdc[15891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:55:54.135424 16408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:54.118815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52876 (local address 127.0.97.2:46295)
0620 02:55:54.119006 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:54.119013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:54.119175 (+ 162us) server_negotiation.cc:407] Connection header received
0620 02:55:54.119411 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:54.119417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:54.119507 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:54.119632 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:54.121269 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:54.122019 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:54.123691 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:54.124162 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:54.124242 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:54.129185 (+ 4943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:54.129200 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:54.129215 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:54.129251 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:54.132357 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:54.133113 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:54.133120 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:54.133123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:54.133255 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:54.133737 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:54.133741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:54.133743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:54.134171 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:54.134461 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:54.134856 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 02:55:54.135069 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 02:55:54.136291 16254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:55:54.173861 16409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:54.173331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40417 (local address 127.0.0.1:41852)
0620 02:55:54.173585 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:54.173692 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260620 02:55:54.175091 16409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:54.174706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40417 (local address 127.0.0.1:41864)
0620 02:55:54.174861 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:54.174954 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 02:55:54.227999 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 16131
I20260620 02:55:54.235812 16409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:54.235267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40417 (local address 127.0.0.1:41870)
0620 02:55:54.235534 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:54.235629 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 02:55:54.237257 16409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:54.236619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40417 (local address 127.0.0.1:41872)
0620 02:55:54.236802 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:54.237134 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 02:55:54.245693 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 16264
Jun 20 02:55:54 dist-test-slave-8p8c krb5kdc[15891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:55:54.261696 16405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:54.246564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39058 (local address 127.0.97.3:45263)
0620 02:55:54.246793 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 02:55:54.246799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:55:54.246821 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:55:54.247025 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:55:54.247035 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:55:54.247105 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:55:54.247221 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:55:54.248563 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:54.249317 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:54.250842 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:55:54.251241 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:55:54.251378 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:55:54.256006 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:55:54.256021 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:55:54.256038 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:55:54.256088 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:55:54.258880 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:54.259603 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:54.259611 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:54.259613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:54.259740 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:55:54.260147 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:55:54.260157 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:55:54.260159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:55:54.260503 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:55:54.260804 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0620 02:55:54.261127 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 02:55:54.261353 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 02:55:54.262323 16387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:55:54.317942 16409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:54.317446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40417 (local address 127.0.0.1:41884)
0620 02:55:54.317680 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:54.317786 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 02:55:54.319233 16409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:55:54.318792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40417 (local address 127.0.0.1:41886)
0620 02:55:54.319002 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 02:55:54.319080 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260620 02:55:54.354064 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 16264
I20260620 02:55:54.373313 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 15904
I20260620 02:55:54.374536 15966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:55:54.492339 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 15904
I20260620 02:56:14.514798 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34103
--webserver_interface=127.0.97.62
--webserver_port=43851
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:56:14.861301 16442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:56:14.861704 16442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:56:14.861888 16442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:56:14.869275 16442 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 02:56:14.869428 16442 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 02:56:14.869515 16442 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 02:56:14.869621 16442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:56:14.869715 16442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:56:14.869825 16442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:56:14.869930 16442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:56:14.870019 16442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:56:14.880419 16442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=43851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:56:14.881893 16442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:56:14.883440 16442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:56:14.907348 16449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:56:14.907380 16451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:56:14.907429 16448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:56:14.908190 16442 server_base.cc:1034] running on GCE node
I20260620 02:56:14.909186 16442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:56:14.910943 16442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:56:14.912241 16442 hybrid_clock.cc:648] HybridClock initialized: now 1781924174912150 us; error 67 us; skew 500 ppm
I20260620 02:56:14.912693 16442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:56:14 dist-test-slave-8p8c krb5kdc[15891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:56:14.916500 16442 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:56:14.918208 16442 webserver.cc:460] Webserver started at http://127.0.97.62:43851/ using document root <none> and password file <none>
I20260620 02:56:14.919044 16442 fs_manager.cc:359] Metadata directory not provided
I20260620 02:56:14.919153 16442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:56:14.925407 16442 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 02:56:14.929162 16458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:56:14.930970 16442 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260620 02:56:14.931151 16442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "0c70203eb93e498c94be7647fcb3ab45"
format_stamp: "Formatted at 2026-06-20 02:55:51 on dist-test-slave-8p8c"
server_key: "60105ab7a1ceaa2b659f095cbec104e4"
server_key_iv: "a98a41bdd6b08775fca4bb39eb4c8c85"
server_key_version: "encryptionkey@0"
I20260620 02:56:14.932282 16442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:56:14.947705 16442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:56:14.952200 16442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:56:14.952459 16442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:56:14.974071 16442 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34103
I20260620 02:56:14.974105 16509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34103 every 8 connection(s)
I20260620 02:56:14.975812 16442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:56:14.980341 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 16442
I20260620 02:56:14.981359 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:40417
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:34103
--webserver_port=45737
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:56:14.985153 16510 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:56:14.990718 16510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45: Bootstrap starting.
I20260620 02:56:15.011384 16510 log.cc:824] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45: Log is configured to *not* fsync() on all Append() calls
I20260620 02:56:15.024564 16510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:56:15.025192 16510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45: Bootstrap complete.
I20260620 02:56:15.035019 16510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c70203eb93e498c94be7647fcb3ab45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34103 } }
I20260620 02:56:15.035976 16510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c70203eb93e498c94be7647fcb3ab45, State: Initialized, Role: FOLLOWER
I20260620 02:56:15.036906 16510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c70203eb93e498c94be7647fcb3ab45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34103 } }
I20260620 02:56:15.037184 16510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:56:15.037331 16510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:56:15.037560 16510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [term 1 FOLLOWER]: Advancing to term 2
I20260620 02:56:15.041851 16510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c70203eb93e498c94be7647fcb3ab45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34103 } }
I20260620 02:56:15.042465 16510 leader_election.cc:302] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c70203eb93e498c94be7647fcb3ab45; no voters:
I20260620 02:56:15.043109 16510 leader_election.cc:288] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 02:56:15.043311 16514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 02:56:15.044703 16514 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [term 2 LEADER]: Becoming Leader. State: Replica: 0c70203eb93e498c94be7647fcb3ab45, State: Running, Role: LEADER
I20260620 02:56:15.045356 16510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:56:15.045310 16514 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c70203eb93e498c94be7647fcb3ab45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34103 } }
I20260620 02:56:15.050251 16516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c70203eb93e498c94be7647fcb3ab45. Latest consensus state: current_term: 2 leader_uuid: "0c70203eb93e498c94be7647fcb3ab45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c70203eb93e498c94be7647fcb3ab45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34103 } } }
I20260620 02:56:15.050318 16515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0c70203eb93e498c94be7647fcb3ab45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c70203eb93e498c94be7647fcb3ab45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34103 } } }
I20260620 02:56:15.050724 16516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [sys.catalog]: This master's current role is: LEADER
I20260620 02:56:15.050796 16515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [sys.catalog]: This master's current role is: LEADER
I20260620 02:56:15.052198 16523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:56:15.065052 16523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:56:15.067056 16523 catalog_manager.cc:1259] Loaded cluster ID: d9a6e47a066448ad9d89372750215479
I20260620 02:56:15.067268 16523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:56:15.072723 16523 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:56:15.074942 16523 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45: Loaded TSK: 0
I20260620 02:56:15.091393 16531 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 02:56:15.108660 16523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45: Generated new TSK 1
I20260620 02:56:15.109077 16523 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 02:56:15.372359 16512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:56:15.372666 16512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:56:15.372820 16512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:56:15.380539 16512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:56:15.380661 16512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:56:15.380838 16512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:56:15.391260 16512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:40417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45737
--tserver_master_addrs=127.0.97.62:34103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:56:15.392625 16512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:56:15.393978 16512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:56:15.409631 16537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:56:15.409618 16538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:56:15.410619 16540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:56:15.410596 16512 server_base.cc:1034] running on GCE node
I20260620 02:56:15.411383 16512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:56:15.412489 16512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:56:15.413719 16512 hybrid_clock.cc:648] HybridClock initialized: now 1781924175413673 us; error 51 us; skew 500 ppm
I20260620 02:56:15.414045 16512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:56:15 dist-test-slave-8p8c krb5kdc[15891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:56:15.417651 16512 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:56:15.419579 16512 webserver.cc:460] Webserver started at http://127.0.97.1:45737/ using document root <none> and password file <none>
I20260620 02:56:15.420481 16512 fs_manager.cc:359] Metadata directory not provided
I20260620 02:56:15.420635 16512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:56:15.427466 16512 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 02:56:15.431020 16547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:56:15.432777 16512 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:56:15.432946 16512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/wal
uuid: "5115eed9ba524de59a14e34a4d3f1593"
format_stamp: "Formatted at 2026-06-20 02:55:52 on dist-test-slave-8p8c"
server_key: "b3b4318230e90b0377a19228a2b3e49d"
server_key_iv: "a420dc8272a8c8bd1d616f5eb996ebb0"
server_key_version: "encryptionkey@0"
I20260620 02:56:15.434336 16512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:56:15.453796 16512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:56:15.459760 16512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:56:15.460057 16512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:56:15.461201 16512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:56:15.463430 16554 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260620 02:56:15.468978 16554 ts_tablet_manager.cc:1852] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260620 02:56:15.470072 16554 ts_tablet_manager.cc:1905] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:56:15.477414 16554 ts_tablet_manager.cc:1918] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:56:15.477746 16554 ts_tablet_manager.cc:1939] T cef70bfe6d8d4d1881c595270a03568e P 5115eed9ba524de59a14e34a4d3f1593: Deleting consensus metadata
I20260620 02:56:15.478153 16512 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 02:56:15.478269 16512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20260620 02:56:15.478446 16512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:56:15.478516 16512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:56:15.515714 16512 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40417
I20260620 02:56:15.515810 16660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40417 every 8 connection(s)
I20260620 02:56:15.517606 16512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:56:15.521112 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 16512
I20260620 02:56:15.521989 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:46295
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:34103
--webserver_port=44397
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:56:15 dist-test-slave-8p8c krb5kdc[15891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:56:15.547266 16663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:15.521168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48889 (local address 127.0.97.62:34103)
0620 02:56:15.521871 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0620 02:56:15.521889 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 02:56:15.522664 (+ 775us) server_negotiation.cc:407] Connection header received
0620 02:56:15.524986 (+ 2322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:56:15.525008 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:56:15.525482 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:56:15.525922 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:56:15.527592 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:15.529756 (+ 2164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:15.532740 (+ 2984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:15.533462 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:15.533640 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:15.538494 (+ 4854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:56:15.538503 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:56:15.538520 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:56:15.538560 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:56:15.541842 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:15.542978 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:15.542986 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:15.542992 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:15.543169 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:15.543761 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:15.543765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:15.543767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:15.544257 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:56:15.544427 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 02:56:15.545669 (+ 1242us) server_negotiation.cc:299] Negotiation successful
0620 02:56:15.546091 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":294,"threads_started":1}
I20260620 02:56:15.547508 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:15.521244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.1:48889)
0620 02:56:15.522445 (+ 1201us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:15.522492 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 02:56:15.524334 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:56:15.526064 (+ 1730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:56:15.526075 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:56:15.526749 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:56:15.527321 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:56:15.527341 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:56:15.530090 (+ 2749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:56:15.530099 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:56:15.532535 (+ 2436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:56:15.532558 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:56:15.533741 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:56:15.533752 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:56:15.534088 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:15.534901 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:56:15.534936 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:56:15.538088 (+ 3152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:56:15.542036 (+ 3948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:56:15.542049 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:56:15.542070 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:56:15.542820 (+ 750us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:56:15.543292 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:56:15.543300 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:56:15.543305 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:56:15.543558 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:56:15.544415 (+ 857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:56:15.544427 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:56:15.544939 (+ 512us) client_negotiation.cc:769] Sending connection context
0620 02:56:15.545311 (+ 372us) client_negotiation.cc:240] Negotiation successful
0620 02:56:15.545974 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"thread_start_us":552,"threads_started":1}
I20260620 02:56:15.550457 16661 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34103
I20260620 02:56:15.550803 16661 heartbeater.cc:464] Registering TS with master...
I20260620 02:56:15.551589 16661 heartbeater.cc:511] Master 127.0.97.62:34103 requested a full tablet report, sending...
I20260620 02:56:15.553706 16475 ts_manager.cc:194] Registered new tserver with Master: 5115eed9ba524de59a14e34a4d3f1593 (127.0.97.1:40417)
I20260620 02:56:15.555224 16475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48889
W20260620 02:56:15.869596 16665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:56:15.869987 16665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:56:15.870178 16665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:56:15.878417 16665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:56:15.878618 16665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:56:15.878839 16665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:56:15.889139 16665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:46295
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=44397
--tserver_master_addrs=127.0.97.62:34103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:56:15.890625 16665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:56:15.892031 16665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:56:15.906598 16671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:56:15.906625 16670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:56:15.906759 16673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:56:15.907022 16665 server_base.cc:1034] running on GCE node
I20260620 02:56:15.907814 16665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:56:15.908763 16665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:56:15.909988 16665 hybrid_clock.cc:648] HybridClock initialized: now 1781924175909947 us; error 31 us; skew 500 ppm
I20260620 02:56:15.910360 16665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:56:15 dist-test-slave-8p8c krb5kdc[15891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:56:15.913875 16665 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:56:15.915757 16665 webserver.cc:460] Webserver started at http://127.0.97.2:44397/ using document root <none> and password file <none>
I20260620 02:56:15.916539 16665 fs_manager.cc:359] Metadata directory not provided
I20260620 02:56:15.916716 16665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:56:15.923174 16665 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:56:15.926182 16680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:56:15.927904 16665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 02:56:15.928133 16665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/wal
uuid: "2f80cb4ae084422183967f584a2b52d0"
format_stamp: "Formatted at 2026-06-20 02:55:53 on dist-test-slave-8p8c"
server_key: "b75a451ad3e3e4f820f050cdd4a12b0d"
server_key_iv: "4039541c37ce32f4bef155f5d7ed0b79"
server_key_version: "encryptionkey@0"
I20260620 02:56:15.929284 16665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:56:15.961206 16665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:56:15.966279 16665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:56:15.966646 16665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:56:15.967788 16665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:56:15.969379 16665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:56:15.969520 16665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:56:15.969828 16665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:56:15.969997 16665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:56:16.015959 16665 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46295
I20260620 02:56:16.015995 16792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46295 every 8 connection(s)
I20260620 02:56:16.017679 16665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:56:16.022130 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 16665
I20260620 02:56:16.022913 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:45263
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:34103
--webserver_port=39989
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 02:56:16 dist-test-slave-8p8c krb5kdc[15891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:56:16.045073 16663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:16.020899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53797 (local address 127.0.97.62:34103)
0620 02:56:16.021181 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 02:56:16.021186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:56:16.022643 (+ 1457us) server_negotiation.cc:407] Connection header received
0620 02:56:16.025683 (+ 3040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:56:16.025688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:56:16.025778 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:56:16.025874 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:56:16.028265 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.029039 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.031725 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.032522 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.032684 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:16.037007 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:56:16.037017 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:56:16.037025 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:56:16.037075 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:56:16.041246 (+ 4171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:16.042175 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:16.042184 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:16.042190 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:16.042355 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:16.042849 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:16.042857 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:16.042863 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:16.043278 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:56:16.043446 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 02:56:16.044287 (+ 841us) server_negotiation.cc:299] Negotiation successful
0620 02:56:16.044585 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260620 02:56:16.045933 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:16.021466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.2:53797)
0620 02:56:16.022461 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:16.022514 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 02:56:16.025323 (+ 2809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:56:16.026200 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:56:16.026215 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:56:16.027115 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:56:16.028002 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:56:16.028035 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.029220 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.029228 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:56:16.031528 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:56:16.031551 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.032769 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.032775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:56:16.033019 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:16.033858 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:56:16.033884 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:56:16.036729 (+ 2845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:56:16.041437 (+ 4708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:56:16.041448 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:56:16.041464 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:56:16.042007 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:56:16.042516 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:56:16.042520 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:56:16.042522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:56:16.042720 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:56:16.043439 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:56:16.043446 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:56:16.043830 (+ 384us) client_negotiation.cc:769] Sending connection context
0620 02:56:16.044123 (+ 293us) client_negotiation.cc:240] Negotiation successful
0620 02:56:16.044633 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":393,"threads_started":1}
I20260620 02:56:16.047860 16793 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34103
I20260620 02:56:16.048265 16793 heartbeater.cc:464] Registering TS with master...
I20260620 02:56:16.049497 16793 heartbeater.cc:511] Master 127.0.97.62:34103 requested a full tablet report, sending...
I20260620 02:56:16.051805 16475 ts_manager.cc:194] Registered new tserver with Master: 2f80cb4ae084422183967f584a2b52d0 (127.0.97.2:46295)
I20260620 02:56:16.052755 16475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53797
W20260620 02:56:16.374670 16796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:56:16.374982 16796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:56:16.375176 16796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:56:16.382570 16796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:56:16.382743 16796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:56:16.382959 16796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:56:16.392763 16796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:45263
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=39989
--tserver_master_addrs=127.0.97.62:34103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:56:16.394712 16796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:56:16.396453 16796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:56:16.411011 16804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:56:16.410954 16801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:56:16.410985 16802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:56:16.411840 16796 server_base.cc:1034] running on GCE node
I20260620 02:56:16.412452 16796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:56:16.413467 16796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:56:16.414698 16796 hybrid_clock.cc:648] HybridClock initialized: now 1781924176414628 us; error 61 us; skew 500 ppm
I20260620 02:56:16.414988 16796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:56:16 dist-test-slave-8p8c krb5kdc[15891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:56:16.418336 16796 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:56:16.420048 16796 webserver.cc:460] Webserver started at http://127.0.97.3:39989/ using document root <none> and password file <none>
I20260620 02:56:16.420720 16796 fs_manager.cc:359] Metadata directory not provided
I20260620 02:56:16.420835 16796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:56:16.426646 16796 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 02:56:16.430961 16811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:56:16.433110 16796 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 02:56:16.433302 16796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/wal
uuid: "85367245efeb4a08b1599f5ffcced30c"
format_stamp: "Formatted at 2026-06-20 02:55:53 on dist-test-slave-8p8c"
server_key: "9717a6c83bf6296baa55c03fe57e9bd1"
server_key_iv: "676b9c17c40aa29b562bf1f6073e92b4"
server_key_version: "encryptionkey@0"
I20260620 02:56:16.434796 16796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:56:16.450124 16796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:56:16.455665 16796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:56:16.455928 16796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:56:16.456980 16796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:56:16.458616 16796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:56:16.458727 16796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:56:16.458807 16796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:56:16.458866 16796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:56:16.495249 16796 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45263
I20260620 02:56:16.495453 16923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45263 every 8 connection(s)
I20260620 02:56:16.496833 16796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:56:16.504446 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 16796
Jun 20 02:56:16 dist-test-slave-8p8c krb5kdc[15891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:56:16.519979 16663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:16.499743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60951 (local address 127.0.97.62:34103)
0620 02:56:16.500016 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 02:56:16.500022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:56:16.501312 (+ 1290us) server_negotiation.cc:407] Connection header received
0620 02:56:16.503226 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:56:16.503232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:56:16.503324 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:56:16.503438 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:56:16.505439 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.506437 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.508006 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.508423 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.508566 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:16.512615 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:56:16.512624 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:56:16.512633 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:56:16.512684 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:56:16.516298 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:16.517133 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:16.517144 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:16.517150 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:16.517298 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:16.517836 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:16.517841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:16.517844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:16.518207 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:56:16.518384 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 02:56:16.519245 (+ 861us) server_negotiation.cc:299] Negotiation successful
0620 02:56:16.519582 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 02:56:16.521188 16926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:16.500308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.3:60951)
0620 02:56:16.501141 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:16.501177 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 02:56:16.502947 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:56:16.503674 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:56:16.503682 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:56:16.504472 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:56:16.505252 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:56:16.505278 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.506603 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.506608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:56:16.507855 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:56:16.507866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.508645 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.508655 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:56:16.508915 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:16.509614 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:56:16.509639 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:56:16.512318 (+ 2679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:56:16.516470 (+ 4152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:56:16.516483 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:56:16.516502 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:56:16.516977 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:56:16.517430 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:56:16.517437 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:56:16.517442 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:56:16.517704 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:56:16.518440 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:56:16.518451 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:56:16.518797 (+ 346us) client_negotiation.cc:769] Sending connection context
0620 02:56:16.519138 (+ 341us) client_negotiation.cc:240] Negotiation successful
0620 02:56:16.519845 (+ 707us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":353,"threads_started":1}
I20260620 02:56:16.523036 16924 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34103
I20260620 02:56:16.523409 16924 heartbeater.cc:464] Registering TS with master...
I20260620 02:56:16.524329 16924 heartbeater.cc:511] Master 127.0.97.62:34103 requested a full tablet report, sending...
I20260620 02:56:16.524762 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:16.505459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37930 (local address 127.0.97.62:34103)
0620 02:56:16.505972 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0620 02:56:16.505977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:56:16.506485 (+ 508us) server_negotiation.cc:407] Connection header received
0620 02:56:16.506699 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:56:16.506705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:56:16.506765 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:56:16.506887 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:56:16.508507 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.509368 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.511160 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.511738 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.511882 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:16.515784 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:56:16.515790 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:56:16.515794 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:56:16.515843 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:56:16.520119 (+ 4276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:16.522416 (+ 2297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:16.522425 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:16.522430 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:16.522579 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:16.523070 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:16.523076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:16.523079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:16.523506 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:56:16.523786 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 02:56:16.524114 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 02:56:16.524362 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":280,"threads_started":1}
I20260620 02:56:16.526000 16475 ts_manager.cc:194] Registered new tserver with Master: 85367245efeb4a08b1599f5ffcced30c (127.0.97.3:45263)
I20260620 02:56:16.527269 16475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60951
I20260620 02:56:16.527865 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:56:16.536262 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:16.529112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37944 (local address 127.0.97.62:34103)
0620 02:56:16.529370 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 02:56:16.529377 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:56:16.529710 (+ 333us) server_negotiation.cc:407] Connection header received
0620 02:56:16.529900 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:56:16.529905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:56:16.530091 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:56:16.530270 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:56:16.531168 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.532155 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.533790 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.534410 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.534545 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:16.535115 (+ 570us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:56:16.535364 (+ 249us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 02:56:16.535872 (+ 508us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37944: token expired
Metrics: {"server-negotiator.queue_time_us":89}
W20260620 02:56:16.536408 16927 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37944: token expired
W20260620 02:56:16.537322 16397 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:34103)
I20260620 02:56:16.554836 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:16.537848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37952 (local address 127.0.97.62:34103)
0620 02:56:16.538064 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 02:56:16.538070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:56:16.538091 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:56:16.538210 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:56:16.538217 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:56:16.538298 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:56:16.538463 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:56:16.540682 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.541477 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.543881 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.544304 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.544409 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:16.547690 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:56:16.547729 (+ 39us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:56:16.547737 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:56:16.547791 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:56:16.551583 (+ 3792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:16.552484 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:16.552494 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:16.552498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:16.552672 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:16.553134 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:16.553140 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:16.553142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:16.553496 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:56:16.553782 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0620 02:56:16.554148 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 02:56:16.554438 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 02:56:16.558367 16661 heartbeater.cc:503] Master 127.0.97.62:34103 was elected leader, sending a full tablet report...
I20260620 02:56:16.558899 16475 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:56:16.566524 16475 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:56:16.591809 16938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:16.582348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46295 (local address 127.0.0.1:48066)
0620 02:56:16.583620 (+ 1272us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:16.583640 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:56:16.583770 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:56:16.584707 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:56:16.584711 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:56:16.584735 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:56:16.585100 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:56:16.585113 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.586920 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.586925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:56:16.589189 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:56:16.589204 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.590719 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.590725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:56:16.590909 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:16.590971 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 02:56:16.591132 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 02:56:16.591329 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1133,"mutex_wait_us":87,"thread_start_us":959,"threads_started":1}
I20260620 02:56:16.591974 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:16.579775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44534 (local address 127.0.97.3:45263)
0620 02:56:16.582085 (+ 2310us) server_negotiation.cc:206] Beginning negotiation
0620 02:56:16.583518 (+ 1433us) server_negotiation.cc:399] Waiting for connection header
0620 02:56:16.583560 (+ 42us) server_negotiation.cc:407] Connection header received
0620 02:56:16.583725 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:56:16.583731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:56:16.583983 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:56:16.584780 (+ 797us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:56:16.585041 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.586650 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.589485 (+ 2835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.590866 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.591058 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:16.591278 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 02:56:16.591504 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 02:56:16.591667 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2132,"thread_start_us":1995,"threads_started":1}
I20260620 02:56:16.592227 16934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:16.579576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45263 (local address 127.0.0.1:44534)
0620 02:56:16.581126 (+ 1550us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:16.581162 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 02:56:16.581332 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:56:16.584195 (+ 2863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:56:16.584203 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:56:16.584232 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:56:16.584614 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:56:16.584629 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.586819 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.586825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:56:16.589211 (+ 2386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:56:16.589224 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.591129 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.591135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:56:16.591313 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:16.591345 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 02:56:16.591465 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 02:56:16.591611 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1298,"mutex_wait_us":139,"thread_start_us":1145,"threads_started":1}
I20260620 02:56:16.592852 16939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:16.582764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48066 (local address 127.0.97.2:46295)
0620 02:56:16.583264 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0620 02:56:16.583274 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:56:16.583926 (+ 652us) server_negotiation.cc:407] Connection header received
0620 02:56:16.584043 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:56:16.584051 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:56:16.584269 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:56:16.584461 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:56:16.585290 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.586784 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.589453 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.590518 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.590666 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:16.590890 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 02:56:16.592266 (+ 1376us) server_negotiation.cc:299] Negotiation successful
0620 02:56:16.592563 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":257,"threads_started":1}
I20260620 02:56:16.593173 16935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:16.579655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40417 (local address 127.0.0.1:58936)
0620 02:56:16.581912 (+ 2257us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:16.581933 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:56:16.582095 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:56:16.583487 (+ 1392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:56:16.583495 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:56:16.583538 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:56:16.584276 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:56:16.584295 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.586085 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.586093 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:56:16.588798 (+ 2705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:56:16.588813 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.591043 (+ 2230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.591049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:56:16.591223 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:16.591253 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 02:56:16.591327 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 02:56:16.591439 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2131,"mutex_wait_us":224,"thread_start_us":2521,"threads_started":1}
I20260620 02:56:16.593730 16937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:16.580233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58936 (local address 127.0.97.1:40417)
0620 02:56:16.582661 (+ 2428us) server_negotiation.cc:206] Beginning negotiation
0620 02:56:16.582674 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:56:16.582713 (+ 39us) server_negotiation.cc:407] Connection header received
0620 02:56:16.582862 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:56:16.582874 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:56:16.583213 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:56:16.583443 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:56:16.584545 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.585949 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.588963 (+ 3014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.590587 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.590714 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:16.590943 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 02:56:16.591727 (+ 784us) server_negotiation.cc:299] Negotiation successful
0620 02:56:16.593408 (+ 1681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2219,"thread_start_us":2084,"threads_started":1}
I20260620 02:56:16.595366 16596 tablet_service.cc:1467] Processing CreateTablet for tablet 72575507a19c4ba683dbd8d28a3ae5a9 (DEFAULT_TABLE table=test-table [id=9c493bc0dd1f448aa2b63e907d30a3a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:56:16.596534 16596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72575507a19c4ba683dbd8d28a3ae5a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:56:16.596194 16859 tablet_service.cc:1467] Processing CreateTablet for tablet 72575507a19c4ba683dbd8d28a3ae5a9 (DEFAULT_TABLE table=test-table [id=9c493bc0dd1f448aa2b63e907d30a3a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:56:16.596194 16728 tablet_service.cc:1467] Processing CreateTablet for tablet 72575507a19c4ba683dbd8d28a3ae5a9 (DEFAULT_TABLE table=test-table [id=9c493bc0dd1f448aa2b63e907d30a3a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:56:16.597596 16859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72575507a19c4ba683dbd8d28a3ae5a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:56:16.599550 16728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72575507a19c4ba683dbd8d28a3ae5a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:56:16.608078 16940 tablet_bootstrap.cc:492] T 72575507a19c4ba683dbd8d28a3ae5a9 P 5115eed9ba524de59a14e34a4d3f1593: Bootstrap starting.
I20260620 02:56:16.611891 16940 tablet_bootstrap.cc:654] T 72575507a19c4ba683dbd8d28a3ae5a9 P 5115eed9ba524de59a14e34a4d3f1593: Neither blocks nor log segments found. Creating new log.
I20260620 02:56:16.613127 16941 tablet_bootstrap.cc:492] T 72575507a19c4ba683dbd8d28a3ae5a9 P 85367245efeb4a08b1599f5ffcced30c: Bootstrap starting.
I20260620 02:56:16.614495 16942 tablet_bootstrap.cc:492] T 72575507a19c4ba683dbd8d28a3ae5a9 P 2f80cb4ae084422183967f584a2b52d0: Bootstrap starting.
I20260620 02:56:16.614725 16940 log.cc:824] T 72575507a19c4ba683dbd8d28a3ae5a9 P 5115eed9ba524de59a14e34a4d3f1593: Log is configured to *not* fsync() on all Append() calls
I20260620 02:56:16.617178 16941 tablet_bootstrap.cc:654] T 72575507a19c4ba683dbd8d28a3ae5a9 P 85367245efeb4a08b1599f5ffcced30c: Neither blocks nor log segments found. Creating new log.
I20260620 02:56:16.617641 16942 tablet_bootstrap.cc:654] T 72575507a19c4ba683dbd8d28a3ae5a9 P 2f80cb4ae084422183967f584a2b52d0: Neither blocks nor log segments found. Creating new log.
I20260620 02:56:16.618086 16940 tablet_bootstrap.cc:492] T 72575507a19c4ba683dbd8d28a3ae5a9 P 5115eed9ba524de59a14e34a4d3f1593: No bootstrap required, opened a new log
I20260620 02:56:16.618328 16941 log.cc:824] T 72575507a19c4ba683dbd8d28a3ae5a9 P 85367245efeb4a08b1599f5ffcced30c: Log is configured to *not* fsync() on all Append() calls
I20260620 02:56:16.618353 16940 ts_tablet_manager.cc:1397] T 72575507a19c4ba683dbd8d28a3ae5a9 P 5115eed9ba524de59a14e34a4d3f1593: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260620 02:56:16.618907 16942 log.cc:824] T 72575507a19c4ba683dbd8d28a3ae5a9 P 2f80cb4ae084422183967f584a2b52d0: Log is configured to *not* fsync() on all Append() calls
I20260620 02:56:16.620972 16941 tablet_bootstrap.cc:492] T 72575507a19c4ba683dbd8d28a3ae5a9 P 85367245efeb4a08b1599f5ffcced30c: No bootstrap required, opened a new log
I20260620 02:56:16.621270 16941 ts_tablet_manager.cc:1397] T 72575507a19c4ba683dbd8d28a3ae5a9 P 85367245efeb4a08b1599f5ffcced30c: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260620 02:56:16.623523 16942 tablet_bootstrap.cc:492] T 72575507a19c4ba683dbd8d28a3ae5a9 P 2f80cb4ae084422183967f584a2b52d0: No bootstrap required, opened a new log
I20260620 02:56:16.623785 16942 ts_tablet_manager.cc:1397] T 72575507a19c4ba683dbd8d28a3ae5a9 P 2f80cb4ae084422183967f584a2b52d0: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260620 02:56:16.628734 16941 raft_consensus.cc:348] T 72575507a19c4ba683dbd8d28a3ae5a9 P 85367245efeb4a08b1599f5ffcced30c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85367245efeb4a08b1599f5ffcced30c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45263 } } peers { permanent_uuid: "5115eed9ba524de59a14e34a4d3f1593" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40417 } } peers { permanent_uuid: "2f80cb4ae084422183967f584a2b52d0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46295 } }
I20260620 02:56:16.629149 16941 raft_consensus.cc:374] T 72575507a19c4ba683dbd8d28a3ae5a9 P 85367245efeb4a08b1599f5ffcced30c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:56:16.629241 16941 raft_consensus.cc:729] T 72575507a19c4ba683dbd8d28a3ae5a9 P 85367245efeb4a08b1599f5ffcced30c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85367245efeb4a08b1599f5ffcced30c, State: Initialized, Role: FOLLOWER
I20260620 02:56:16.629362 16940 raft_consensus.cc:348] T 72575507a19c4ba683dbd8d28a3ae5a9 P 5115eed9ba524de59a14e34a4d3f1593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85367245efeb4a08b1599f5ffcced30c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45263 } } peers { permanent_uuid: "5115eed9ba524de59a14e34a4d3f1593" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40417 } } peers { permanent_uuid: "2f80cb4ae084422183967f584a2b52d0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46295 } }
I20260620 02:56:16.629799 16940 raft_consensus.cc:374] T 72575507a19c4ba683dbd8d28a3ae5a9 P 5115eed9ba524de59a14e34a4d3f1593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:56:16.629890 16940 raft_consensus.cc:729] T 72575507a19c4ba683dbd8d28a3ae5a9 P 5115eed9ba524de59a14e34a4d3f1593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5115eed9ba524de59a14e34a4d3f1593, State: Initialized, Role: FOLLOWER
I20260620 02:56:16.630024 16941 consensus_queue.cc:260] T 72575507a19c4ba683dbd8d28a3ae5a9 P 85367245efeb4a08b1599f5ffcced30c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85367245efeb4a08b1599f5ffcced30c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45263 } } peers { permanent_uuid: "5115eed9ba524de59a14e34a4d3f1593" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40417 } } peers { permanent_uuid: "2f80cb4ae084422183967f584a2b52d0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46295 } }
I20260620 02:56:16.630481 16940 consensus_queue.cc:260] T 72575507a19c4ba683dbd8d28a3ae5a9 P 5115eed9ba524de59a14e34a4d3f1593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85367245efeb4a08b1599f5ffcced30c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45263 } } peers { permanent_uuid: "5115eed9ba524de59a14e34a4d3f1593" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40417 } } peers { permanent_uuid: "2f80cb4ae084422183967f584a2b52d0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46295 } }
I20260620 02:56:16.631299 16924 heartbeater.cc:503] Master 127.0.97.62:34103 was elected leader, sending a full tablet report...
I20260620 02:56:16.632022 16941 ts_tablet_manager.cc:1428] T 72575507a19c4ba683dbd8d28a3ae5a9 P 85367245efeb4a08b1599f5ffcced30c: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260620 02:56:16.633872 16940 ts_tablet_manager.cc:1428] T 72575507a19c4ba683dbd8d28a3ae5a9 P 5115eed9ba524de59a14e34a4d3f1593: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
I20260620 02:56:16.636070 16942 raft_consensus.cc:348] T 72575507a19c4ba683dbd8d28a3ae5a9 P 2f80cb4ae084422183967f584a2b52d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85367245efeb4a08b1599f5ffcced30c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45263 } } peers { permanent_uuid: "5115eed9ba524de59a14e34a4d3f1593" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40417 } } peers { permanent_uuid: "2f80cb4ae084422183967f584a2b52d0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46295 } }
I20260620 02:56:16.636619 16942 raft_consensus.cc:374] T 72575507a19c4ba683dbd8d28a3ae5a9 P 2f80cb4ae084422183967f584a2b52d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:56:16.636741 16942 raft_consensus.cc:729] T 72575507a19c4ba683dbd8d28a3ae5a9 P 2f80cb4ae084422183967f584a2b52d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f80cb4ae084422183967f584a2b52d0, State: Initialized, Role: FOLLOWER
I20260620 02:56:16.637542 16942 consensus_queue.cc:260] T 72575507a19c4ba683dbd8d28a3ae5a9 P 2f80cb4ae084422183967f584a2b52d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85367245efeb4a08b1599f5ffcced30c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45263 } } peers { permanent_uuid: "5115eed9ba524de59a14e34a4d3f1593" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40417 } } peers { permanent_uuid: "2f80cb4ae084422183967f584a2b52d0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46295 } }
I20260620 02:56:16.638991 16793 heartbeater.cc:503] Master 127.0.97.62:34103 was elected leader, sending a full tablet report...
I20260620 02:56:16.640007 16942 ts_tablet_manager.cc:1428] T 72575507a19c4ba683dbd8d28a3ae5a9 P 2f80cb4ae084422183967f584a2b52d0: Time spent starting tablet: real 0.016s user 0.013s sys 0.002s
I20260620 02:56:16.662904 16948 raft_consensus.cc:482] T 72575507a19c4ba683dbd8d28a3ae5a9 P 2f80cb4ae084422183967f584a2b52d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:56:16.663343 16948 raft_consensus.cc:504] T 72575507a19c4ba683dbd8d28a3ae5a9 P 2f80cb4ae084422183967f584a2b52d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85367245efeb4a08b1599f5ffcced30c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45263 } } peers { permanent_uuid: "5115eed9ba524de59a14e34a4d3f1593" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40417 } } peers { permanent_uuid: "2f80cb4ae084422183967f584a2b52d0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46295 } }
I20260620 02:56:16.665163 16948 leader_election.cc:288] T 72575507a19c4ba683dbd8d28a3ae5a9 P 2f80cb4ae084422183967f584a2b52d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85367245efeb4a08b1599f5ffcced30c (127.0.97.3:45263), 5115eed9ba524de59a14e34a4d3f1593 (127.0.97.1:40417)
I20260620 02:56:16.672197 16937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:16.665650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43887 (local address 127.0.97.1:40417)
0620 02:56:16.665876 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 02:56:16.665885 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:56:16.666766 (+ 881us) server_negotiation.cc:407] Connection header received
0620 02:56:16.666963 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:56:16.666974 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:56:16.667037 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:56:16.667135 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:56:16.667841 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.668544 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.670342 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.671242 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.671384 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:16.671497 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 02:56:16.671823 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 02:56:16.671948 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 02:56:16.672426 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:16.665619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37687 (local address 127.0.97.3:45263)
0620 02:56:16.665852 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 02:56:16.665861 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:56:16.666667 (+ 806us) server_negotiation.cc:407] Connection header received
0620 02:56:16.666879 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:56:16.666883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:56:16.666947 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:56:16.667055 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:56:16.667780 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.668567 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.670500 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.671362 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.671526 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:16.671660 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 02:56:16.672012 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 02:56:16.672150 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 02:56:16.672691 16949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:16.665499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45263 (local address 127.0.97.2:37687)
0620 02:56:16.666493 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:16.666515 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:56:16.666696 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:56:16.667124 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:56:16.667130 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:56:16.667157 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:56:16.667599 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:56:16.667614 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.668724 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.668729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:56:16.670308 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:56:16.670320 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.671598 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.671605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:56:16.671845 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:16.671871 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 02:56:16.671983 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 02:56:16.672130 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":55,"thread_start_us":550,"threads_started":1}
I20260620 02:56:16.672717 16950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:16.665761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40417 (local address 127.0.97.2:43887)
0620 02:56:16.666647 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:16.666681 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 02:56:16.666814 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:56:16.667147 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:56:16.667151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:56:16.667173 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:56:16.667649 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:56:16.667657 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.668669 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.668677 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:56:16.670171 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:56:16.670183 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.671459 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.671465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:56:16.671659 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:16.671684 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 02:56:16.671799 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 02:56:16.671928 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"mutex_wait_us":25,"thread_start_us":460,"threads_started":1}
I20260620 02:56:16.673796 16879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72575507a19c4ba683dbd8d28a3ae5a9" candidate_uuid: "2f80cb4ae084422183967f584a2b52d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85367245efeb4a08b1599f5ffcced30c" is_pre_election: true
I20260620 02:56:16.674288 16879 raft_consensus.cc:2454] T 72575507a19c4ba683dbd8d28a3ae5a9 P 85367245efeb4a08b1599f5ffcced30c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f80cb4ae084422183967f584a2b52d0 in term 0.
I20260620 02:56:16.674942 16616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72575507a19c4ba683dbd8d28a3ae5a9" candidate_uuid: "2f80cb4ae084422183967f584a2b52d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5115eed9ba524de59a14e34a4d3f1593" is_pre_election: true
I20260620 02:56:16.675117 16682 leader_election.cc:302] T 72575507a19c4ba683dbd8d28a3ae5a9 P 2f80cb4ae084422183967f584a2b52d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f80cb4ae084422183967f584a2b52d0, 85367245efeb4a08b1599f5ffcced30c; no voters:
I20260620 02:56:16.675426 16616 raft_consensus.cc:2454] T 72575507a19c4ba683dbd8d28a3ae5a9 P 5115eed9ba524de59a14e34a4d3f1593 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f80cb4ae084422183967f584a2b52d0 in term 0.
I20260620 02:56:16.675590 16948 raft_consensus.cc:2781] T 72575507a19c4ba683dbd8d28a3ae5a9 P 2f80cb4ae084422183967f584a2b52d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:56:16.675753 16948 raft_consensus.cc:482] T 72575507a19c4ba683dbd8d28a3ae5a9 P 2f80cb4ae084422183967f584a2b52d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:56:16.675868 16948 raft_consensus.cc:3037] T 72575507a19c4ba683dbd8d28a3ae5a9 P 2f80cb4ae084422183967f584a2b52d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:56:16.678122 16948 raft_consensus.cc:504] T 72575507a19c4ba683dbd8d28a3ae5a9 P 2f80cb4ae084422183967f584a2b52d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85367245efeb4a08b1599f5ffcced30c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45263 } } peers { permanent_uuid: "5115eed9ba524de59a14e34a4d3f1593" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40417 } } peers { permanent_uuid: "2f80cb4ae084422183967f584a2b52d0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46295 } }
I20260620 02:56:16.678784 16948 leader_election.cc:288] T 72575507a19c4ba683dbd8d28a3ae5a9 P 2f80cb4ae084422183967f584a2b52d0 [CANDIDATE]: Term 1 election: Requested vote from peers 85367245efeb4a08b1599f5ffcced30c (127.0.97.3:45263), 5115eed9ba524de59a14e34a4d3f1593 (127.0.97.1:40417)
I20260620 02:56:16.679227 16879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72575507a19c4ba683dbd8d28a3ae5a9" candidate_uuid: "2f80cb4ae084422183967f584a2b52d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85367245efeb4a08b1599f5ffcced30c"
I20260620 02:56:16.679281 16616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72575507a19c4ba683dbd8d28a3ae5a9" candidate_uuid: "2f80cb4ae084422183967f584a2b52d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5115eed9ba524de59a14e34a4d3f1593"
I20260620 02:56:16.679420 16879 raft_consensus.cc:3037] T 72575507a19c4ba683dbd8d28a3ae5a9 P 85367245efeb4a08b1599f5ffcced30c [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:56:16.679608 16616 raft_consensus.cc:3037] T 72575507a19c4ba683dbd8d28a3ae5a9 P 5115eed9ba524de59a14e34a4d3f1593 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:56:16.681679 16879 raft_consensus.cc:2454] T 72575507a19c4ba683dbd8d28a3ae5a9 P 85367245efeb4a08b1599f5ffcced30c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f80cb4ae084422183967f584a2b52d0 in term 1.
I20260620 02:56:16.682096 16682 leader_election.cc:302] T 72575507a19c4ba683dbd8d28a3ae5a9 P 2f80cb4ae084422183967f584a2b52d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f80cb4ae084422183967f584a2b52d0, 85367245efeb4a08b1599f5ffcced30c; no voters:
I20260620 02:56:16.682227 16616 raft_consensus.cc:2454] T 72575507a19c4ba683dbd8d28a3ae5a9 P 5115eed9ba524de59a14e34a4d3f1593 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f80cb4ae084422183967f584a2b52d0 in term 1.
I20260620 02:56:16.682435 16948 raft_consensus.cc:2781] T 72575507a19c4ba683dbd8d28a3ae5a9 P 2f80cb4ae084422183967f584a2b52d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:56:16.682994 16948 raft_consensus.cc:686] T 72575507a19c4ba683dbd8d28a3ae5a9 P 2f80cb4ae084422183967f584a2b52d0 [term 1 LEADER]: Becoming Leader. State: Replica: 2f80cb4ae084422183967f584a2b52d0, State: Running, Role: LEADER
I20260620 02:56:16.683622 16948 consensus_queue.cc:237] T 72575507a19c4ba683dbd8d28a3ae5a9 P 2f80cb4ae084422183967f584a2b52d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85367245efeb4a08b1599f5ffcced30c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45263 } } peers { permanent_uuid: "5115eed9ba524de59a14e34a4d3f1593" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40417 } } peers { permanent_uuid: "2f80cb4ae084422183967f584a2b52d0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46295 } }
I20260620 02:56:16.688261 16475 catalog_manager.cc:5495] T 72575507a19c4ba683dbd8d28a3ae5a9 P 2f80cb4ae084422183967f584a2b52d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f80cb4ae084422183967f584a2b52d0 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "2f80cb4ae084422183967f584a2b52d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85367245efeb4a08b1599f5ffcced30c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5115eed9ba524de59a14e34a4d3f1593" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f80cb4ae084422183967f584a2b52d0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46295 } health_report { overall_health: HEALTHY } } }
I20260620 02:56:16.720400 16939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:16.714899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48080 (local address 127.0.97.2:46295)
0620 02:56:16.715157 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 02:56:16.715163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:56:16.715211 (+ 48us) server_negotiation.cc:407] Connection header received
0620 02:56:16.715470 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:56:16.715474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:56:16.715572 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:56:16.715728 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:56:16.716095 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.716817 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.718169 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.718646 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.718794 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:16.719283 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:56:16.719782 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:56:16.719953 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 02:56:16.720038 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 02:56:16.720119 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 02:56:16.728828 16616 raft_consensus.cc:1261] T 72575507a19c4ba683dbd8d28a3ae5a9 P 5115eed9ba524de59a14e34a4d3f1593 [term 1 FOLLOWER]: Refusing update from remote peer 2f80cb4ae084422183967f584a2b52d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:56:16.728847 16879 raft_consensus.cc:1261] T 72575507a19c4ba683dbd8d28a3ae5a9 P 85367245efeb4a08b1599f5ffcced30c [term 1 FOLLOWER]: Refusing update from remote peer 2f80cb4ae084422183967f584a2b52d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:56:16.729919 16951 consensus_queue.cc:1035] T 72575507a19c4ba683dbd8d28a3ae5a9 P 2f80cb4ae084422183967f584a2b52d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5115eed9ba524de59a14e34a4d3f1593" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:56:16.730307 16948 consensus_queue.cc:1035] T 72575507a19c4ba683dbd8d28a3ae5a9 P 2f80cb4ae084422183967f584a2b52d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85367245efeb4a08b1599f5ffcced30c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:56:16.744014 16956 mvcc.cc:205] Tried to move back new op lower bound from 7298761427870720000 to 7298761427700461568. Current Snapshot: MvccSnapshot[applied={T|T < 7298761427870720000}]
I20260620 02:56:16.756771 16475 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:56:16.757093 16475 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:56:16.760345 16475 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45: Sending DeleteTablet for 3 replicas of tablet 72575507a19c4ba683dbd8d28a3ae5a9
I20260620 02:56:16.761713 16728 tablet_service.cc:1514] Processing DeleteTablet for tablet 72575507a19c4ba683dbd8d28a3ae5a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:56:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48066
I20260620 02:56:16.761741 16596 tablet_service.cc:1514] Processing DeleteTablet for tablet 72575507a19c4ba683dbd8d28a3ae5a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:56:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58936
I20260620 02:56:16.761983 16859 tablet_service.cc:1514] Processing DeleteTablet for tablet 72575507a19c4ba683dbd8d28a3ae5a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:56:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44534
I20260620 02:56:16.762811 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 16512
I20260620 02:56:16.775523 16963 tablet_replica.cc:330] T 72575507a19c4ba683dbd8d28a3ae5a9 P 5115eed9ba524de59a14e34a4d3f1593: stopping tablet replica
I20260620 02:56:16.776278 16963 raft_consensus.cc:2229] T 72575507a19c4ba683dbd8d28a3ae5a9 P 5115eed9ba524de59a14e34a4d3f1593 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:56:16.777005 16963 raft_consensus.cc:2258] T 72575507a19c4ba683dbd8d28a3ae5a9 P 5115eed9ba524de59a14e34a4d3f1593 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:56:16.779254 16963 ts_tablet_manager.cc:1905] T 72575507a19c4ba683dbd8d28a3ae5a9 P 5115eed9ba524de59a14e34a4d3f1593: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:56:16.784358 16937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:16.763777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58940 (local address 127.0.97.1:40417)
0620 02:56:16.764010 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 02:56:16.764018 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:56:16.764043 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:56:16.764127 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:56:16.764134 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:56:16.764263 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:56:16.764400 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:56:16.766151 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.767788 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.771382 (+ 3594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.772418 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.772592 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:16.776108 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:56:16.776122 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:56:16.776142 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:56:16.776190 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:56:16.780326 (+ 4136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:16.781199 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:16.781208 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:16.781213 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:16.781384 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:16.781780 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:16.781784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:16.781788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:16.782271 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:56:16.782608 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0620 02:56:16.783719 (+ 1111us) server_negotiation.cc:299] Negotiation successful
0620 02:56:16.783960 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 02:56:16.785148 16656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:56:16.786962 16964 tablet_replica.cc:330] T 72575507a19c4ba683dbd8d28a3ae5a9 P 2f80cb4ae084422183967f584a2b52d0: stopping tablet replica
I20260620 02:56:16.787130 16965 tablet_replica.cc:330] T 72575507a19c4ba683dbd8d28a3ae5a9 P 85367245efeb4a08b1599f5ffcced30c: stopping tablet replica
I20260620 02:56:16.787461 16964 raft_consensus.cc:2229] T 72575507a19c4ba683dbd8d28a3ae5a9 P 2f80cb4ae084422183967f584a2b52d0 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:56:16.787612 16965 raft_consensus.cc:2229] T 72575507a19c4ba683dbd8d28a3ae5a9 P 85367245efeb4a08b1599f5ffcced30c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:56:16.788182 16965 raft_consensus.cc:2258] T 72575507a19c4ba683dbd8d28a3ae5a9 P 85367245efeb4a08b1599f5ffcced30c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:56:16.788166 16964 raft_consensus.cc:2258] T 72575507a19c4ba683dbd8d28a3ae5a9 P 2f80cb4ae084422183967f584a2b52d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:56:16.790329 16965 ts_tablet_manager.cc:1905] T 72575507a19c4ba683dbd8d28a3ae5a9 P 85367245efeb4a08b1599f5ffcced30c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:56:16.791030 16964 ts_tablet_manager.cc:1905] T 72575507a19c4ba683dbd8d28a3ae5a9 P 2f80cb4ae084422183967f584a2b52d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:56:16.797972 16965 ts_tablet_manager.cc:1918] T 72575507a19c4ba683dbd8d28a3ae5a9 P 85367245efeb4a08b1599f5ffcced30c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:56:16.798278 16965 log.cc:1196] T 72575507a19c4ba683dbd8d28a3ae5a9 P 85367245efeb4a08b1599f5ffcced30c: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/wal/wals/72575507a19c4ba683dbd8d28a3ae5a9
I20260620 02:56:16.799021 16965 ts_tablet_manager.cc:1939] T 72575507a19c4ba683dbd8d28a3ae5a9 P 85367245efeb4a08b1599f5ffcced30c: Deleting consensus metadata
I20260620 02:56:16.799517 16964 ts_tablet_manager.cc:1918] T 72575507a19c4ba683dbd8d28a3ae5a9 P 2f80cb4ae084422183967f584a2b52d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:56:16.799719 16964 log.cc:1196] T 72575507a19c4ba683dbd8d28a3ae5a9 P 2f80cb4ae084422183967f584a2b52d0: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/wal/wals/72575507a19c4ba683dbd8d28a3ae5a9
I20260620 02:56:16.800366 16964 ts_tablet_manager.cc:1939] T 72575507a19c4ba683dbd8d28a3ae5a9 P 2f80cb4ae084422183967f584a2b52d0: Deleting consensus metadata
I20260620 02:56:16.802349 16460 catalog_manager.cc:4841] TS 2f80cb4ae084422183967f584a2b52d0 (127.0.97.2:46295): tablet 72575507a19c4ba683dbd8d28a3ae5a9 (table test-table [id=9c493bc0dd1f448aa2b63e907d30a3a4]) successfully deleted
I20260620 02:56:16.802794 16460 catalog_manager.cc:4841] TS 85367245efeb4a08b1599f5ffcced30c (127.0.97.3:45263): tablet 72575507a19c4ba683dbd8d28a3ae5a9 (table test-table [id=9c493bc0dd1f448aa2b63e907d30a3a4]) successfully deleted
I20260620 02:56:16.893589 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 16512
W20260620 02:56:16.918841 16462 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:40417 (error 108)
I20260620 02:56:16.919044 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 16665
I20260620 02:56:16.921041 16938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:16.920572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40417 (local address 127.0.0.1:58950)
0620 02:56:16.920740 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:16.920850 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 02:56:16.921350 16462 catalog_manager.cc:4584] TS 5115eed9ba524de59a14e34a4d3f1593 (127.0.97.1:40417): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 72575507a19c4ba683dbd8d28a3ae5a9: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40417: connect: Connection refused (error 111)
I20260620 02:56:16.942176 16939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:16.920538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48094 (local address 127.0.97.2:46295)
0620 02:56:16.920700 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 02:56:16.920706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:56:16.920723 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:56:16.920788 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:56:16.920793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:56:16.920882 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:56:16.920986 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:56:16.922857 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.923683 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.925618 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:16.926108 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:16.926215 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:16.934968 (+ 8753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:56:16.934990 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:56:16.935005 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:56:16.935058 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:56:16.938903 (+ 3845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:16.939955 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:16.939963 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:16.939966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:16.940132 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:16.940583 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:16.940593 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:16.940595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:16.941010 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:56:16.941313 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 02:56:16.941617 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 02:56:16.941792 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 02:56:16.942975 16788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:56:16.975533 16938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:16.974900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40417 (local address 127.0.0.1:58954)
0620 02:56:16.975229 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:16.975360 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
I20260620 02:56:16.976794 16938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:16.976394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40417 (local address 127.0.0.1:58966)
0620 02:56:16.976562 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:16.976669 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 02:56:17.037473 16938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:17.036848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40417 (local address 127.0.0.1:58976)
0620 02:56:17.037164 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:17.037277 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260620 02:56:17.038897 16938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:17.038475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40417 (local address 127.0.0.1:58988)
0620 02:56:17.038657 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:17.038747 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 02:56:17.073132 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 16665
I20260620 02:56:17.092628 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 16796
I20260620 02:56:17.107589 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:17.093478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44536 (local address 127.0.97.3:45263)
0620 02:56:17.093685 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 02:56:17.093690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:56:17.093918 (+ 228us) server_negotiation.cc:407] Connection header received
0620 02:56:17.094112 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:56:17.094117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:56:17.094187 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:56:17.094288 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:56:17.095677 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:17.096441 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:17.097727 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:17.098158 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:17.098269 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:17.101164 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:56:17.101180 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:56:17.101197 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:56:17.101234 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:56:17.104385 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:17.105275 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:17.105287 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:17.105291 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:17.105493 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:17.105977 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:17.105982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:17.105984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:17.106454 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:56:17.106714 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 02:56:17.107128 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 02:56:17.107325 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 02:56:17.108314 16919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:56:17.119650 16938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:17.119033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40417 (local address 127.0.0.1:58994)
0620 02:56:17.119283 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:17.119411 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 02:56:17.120848 16938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:17.120335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40417 (local address 127.0.0.1:59006)
0620 02:56:17.120567 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:17.120693 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 02:56:17.199692 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 16796
I20260620 02:56:17.216292 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 16442
I20260620 02:56:17.217617 16505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:56:17.332523 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 16442
I20260620 02:56:37.355324 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34103
--webserver_interface=127.0.97.62
--webserver_port=43851
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:56:37.697710 16971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:56:37.698058 16971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:56:37.698158 16971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:56:37.705611 16971 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 02:56:37.705724 16971 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 02:56:37.705763 16971 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 02:56:37.705807 16971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:56:37.705861 16971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:56:37.705929 16971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:56:37.705973 16971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:56:37.706020 16971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:56:37.716143 16971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=43851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:56:37.717430 16971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:56:37.718894 16971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:56:37.731907 16977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:56:37.731840 16979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:56:37.732446 16976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:56:37.732471 16971 server_base.cc:1034] running on GCE node
I20260620 02:56:37.733496 16971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:56:37.735037 16971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:56:37.736316 16971 hybrid_clock.cc:648] HybridClock initialized: now 1781924197736232 us; error 56 us; skew 500 ppm
I20260620 02:56:37.736676 16971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:56:37 dist-test-slave-8p8c krb5kdc[15891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:56:37.743675 16971 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:56:37.745204 16971 webserver.cc:460] Webserver started at http://127.0.97.62:43851/ using document root <none> and password file <none>
I20260620 02:56:37.745882 16971 fs_manager.cc:359] Metadata directory not provided
I20260620 02:56:37.745977 16971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:56:37.751557 16971 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260620 02:56:37.754341 16986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:56:37.755885 16971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 02:56:37.756029 16971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "0c70203eb93e498c94be7647fcb3ab45"
format_stamp: "Formatted at 2026-06-20 02:55:51 on dist-test-slave-8p8c"
server_key: "60105ab7a1ceaa2b659f095cbec104e4"
server_key_iv: "a98a41bdd6b08775fca4bb39eb4c8c85"
server_key_version: "encryptionkey@0"
I20260620 02:56:37.757027 16971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:56:37.785723 16971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:56:37.790228 16971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:56:37.790545 16971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:56:37.810271 16971 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34103
I20260620 02:56:37.810322 17037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34103 every 8 connection(s)
I20260620 02:56:37.811861 16971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:56:37.819450 17038 sys_catalog.cc:263] Verifying existing consensus state
I20260620 02:56:37.820403 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 16971
I20260620 02:56:37.821431 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:40417
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:34103
--webserver_port=45737
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:56:37.824402 17038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45: Bootstrap starting.
I20260620 02:56:37.846688 17038 log.cc:824] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45: Log is configured to *not* fsync() on all Append() calls
I20260620 02:56:37.863054 17038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 02:56:37.863621 17038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45: Bootstrap complete.
I20260620 02:56:37.873000 17038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c70203eb93e498c94be7647fcb3ab45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34103 } }
I20260620 02:56:37.874261 17038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c70203eb93e498c94be7647fcb3ab45, State: Initialized, Role: FOLLOWER
I20260620 02:56:37.874889 17038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c70203eb93e498c94be7647fcb3ab45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34103 } }
I20260620 02:56:37.875151 17038 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:56:37.875288 17038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:56:37.875475 17038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [term 2 FOLLOWER]: Advancing to term 3
I20260620 02:56:37.879638 17038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c70203eb93e498c94be7647fcb3ab45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34103 } }
I20260620 02:56:37.880203 17038 leader_election.cc:302] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c70203eb93e498c94be7647fcb3ab45; no voters:
I20260620 02:56:37.880898 17038 leader_election.cc:288] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 02:56:37.881114 17042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 02:56:37.882331 17042 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [term 3 LEADER]: Becoming Leader. State: Replica: 0c70203eb93e498c94be7647fcb3ab45, State: Running, Role: LEADER
I20260620 02:56:37.883224 17038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:56:37.883118 17042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c70203eb93e498c94be7647fcb3ab45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34103 } }
I20260620 02:56:37.886756 17044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c70203eb93e498c94be7647fcb3ab45. Latest consensus state: current_term: 3 leader_uuid: "0c70203eb93e498c94be7647fcb3ab45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c70203eb93e498c94be7647fcb3ab45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34103 } } }
I20260620 02:56:37.887729 17044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [sys.catalog]: This master's current role is: LEADER
I20260620 02:56:37.886926 17043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0c70203eb93e498c94be7647fcb3ab45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c70203eb93e498c94be7647fcb3ab45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34103 } } }
I20260620 02:56:37.888448 17043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45 [sys.catalog]: This master's current role is: LEADER
I20260620 02:56:37.892566 17051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:56:37.902014 17051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:56:37.903626 17051 catalog_manager.cc:1259] Loaded cluster ID: d9a6e47a066448ad9d89372750215479
I20260620 02:56:37.903800 17051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:56:37.907991 17051 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:56:37.909689 17051 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45: Loaded TSK: 0
I20260620 02:56:37.909924 17051 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0c70203eb93e498c94be7647fcb3ab45: Loaded TSK: 1
F20260620 02:56:37.911448 17051 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7efcf10fe9ad google::LogMessage::Fail() at ??:0
@ 0x7efcf110188a google::LogMessage::SendToLog() at ??:0
@ 0x7efcf10fe4de google::LogMessage::Flush() at ??:0
@ 0x7efcf10ffa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7efcf4e627d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7efcf4e5173e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7efd06b2d1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
I20260620 02:56:37.940073 17060 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7efd06b2c783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7efd06bb82a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7efd06bb80c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7efd06c025e9 std::function<>::operator()() at ??:0
@ 0x7efd06b30cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7efd06b2ec36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7efd06bb6610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7efd06bb6439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7efcf3d4a345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7efcf3d4d91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7efcf3d4d799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7efcf3d28eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7efcf70026db start_thread at ??:0
@ 0x7efceff9861f clone at ??:0
W20260620 02:56:38.257359 17040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:56:38.257638 17040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:56:38.257735 17040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:56:38.264873 17040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:56:38.264981 17040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:56:38.265097 17040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:56:38.274613 17040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:40417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45737
--tserver_master_addrs=127.0.97.62:34103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:56:38.275904 17040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:56:38.277225 17040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:56:38.290620 17086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:56:38.290719 17088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:56:38.290966 17085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:56:38.291958 17040 server_base.cc:1034] running on GCE node
I20260620 02:56:38.292583 17040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:56:38.293509 17040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:56:38.294725 17040 hybrid_clock.cc:648] HybridClock initialized: now 1781924198294662 us; error 60 us; skew 500 ppm
I20260620 02:56:38.295030 17040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:56:38 dist-test-slave-8p8c krb5kdc[15891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:56:38.298071 17040 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:56:38.299578 17040 webserver.cc:460] Webserver started at http://127.0.97.1:45737/ using document root <none> and password file <none>
I20260620 02:56:38.300274 17040 fs_manager.cc:359] Metadata directory not provided
I20260620 02:56:38.300419 17040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:56:38.305685 17040 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 02:56:38.308672 17095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:56:38.309929 17040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:56:38.310101 17040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/wal
uuid: "5115eed9ba524de59a14e34a4d3f1593"
format_stamp: "Formatted at 2026-06-20 02:55:52 on dist-test-slave-8p8c"
server_key: "b3b4318230e90b0377a19228a2b3e49d"
server_key_iv: "a420dc8272a8c8bd1d616f5eb996ebb0"
server_key_version: "encryptionkey@0"
I20260620 02:56:38.311129 17040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:56:38.325522 17040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:56:38.330255 17040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:56:38.330590 17040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:56:38.331707 17040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:56:38.333611 17102 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260620 02:56:38.339691 17102 ts_tablet_manager.cc:1852] T 72575507a19c4ba683dbd8d28a3ae5a9 P 5115eed9ba524de59a14e34a4d3f1593: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260620 02:56:38.340653 17102 ts_tablet_manager.cc:1905] T 72575507a19c4ba683dbd8d28a3ae5a9 P 5115eed9ba524de59a14e34a4d3f1593: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:56:38.347776 17102 ts_tablet_manager.cc:1918] T 72575507a19c4ba683dbd8d28a3ae5a9 P 5115eed9ba524de59a14e34a4d3f1593: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:56:38.347970 17102 log.cc:1196] T 72575507a19c4ba683dbd8d28a3ae5a9 P 5115eed9ba524de59a14e34a4d3f1593: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/wal/wals/72575507a19c4ba683dbd8d28a3ae5a9
I20260620 02:56:38.348486 17102 ts_tablet_manager.cc:1939] T 72575507a19c4ba683dbd8d28a3ae5a9 P 5115eed9ba524de59a14e34a4d3f1593: Deleting consensus metadata
I20260620 02:56:38.348978 17040 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 02:56:38.349143 17040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.001s sys 0.000s
I20260620 02:56:38.349313 17040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:56:38.349412 17040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:56:38.385365 17040 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40417
I20260620 02:56:38.385454 17208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40417 every 8 connection(s)
I20260620 02:56:38.386801 17040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:56:38.390020 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 17040
I20260620 02:56:38.390801 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:46295
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:34103
--webserver_port=44397
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:56:38.392510 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:38.390182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.1:47099)
0620 02:56:38.391219 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:38.391493 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":457,"threads_started":1}
W20260620 02:56:38.393302 17209 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34103 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:34103: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
I20260620 02:56:38.394332 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:38.393954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.1:47907)
0620 02:56:38.394142 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:38.394218 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 02:56:38.395556 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:38.395124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.1:55737)
0620 02:56:38.395303 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:38.395399 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260620 02:56:38.395861 17209 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 02:56:38.742442 17212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:56:38.742729 17212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:56:38.742825 17212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:56:38.750085 17212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:56:38.750209 17212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:56:38.750325 17212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:56:38.760557 17212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:46295
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=44397
--tserver_master_addrs=127.0.97.62:34103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:56:38.761896 17212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:56:38.763290 17212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:56:38.776758 17218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:56:38.776958 17217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:56:38.777212 17220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:56:38.777632 17212 server_base.cc:1034] running on GCE node
I20260620 02:56:38.778280 17212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:56:38.779305 17212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:56:38.780593 17212 hybrid_clock.cc:648] HybridClock initialized: now 1781924198780539 us; error 57 us; skew 500 ppm
I20260620 02:56:38.780952 17212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:56:38 dist-test-slave-8p8c krb5kdc[15891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:56:38.784590 17212 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:56:38.786159 17212 webserver.cc:460] Webserver started at http://127.0.97.2:44397/ using document root <none> and password file <none>
I20260620 02:56:38.786972 17212 fs_manager.cc:359] Metadata directory not provided
I20260620 02:56:38.787137 17212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:56:38.793203 17212 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 02:56:38.796267 17227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:56:38.797814 17212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:56:38.798017 17212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/wal
uuid: "2f80cb4ae084422183967f584a2b52d0"
format_stamp: "Formatted at 2026-06-20 02:55:53 on dist-test-slave-8p8c"
server_key: "b75a451ad3e3e4f820f050cdd4a12b0d"
server_key_iv: "4039541c37ce32f4bef155f5d7ed0b79"
server_key_version: "encryptionkey@0"
I20260620 02:56:38.799124 17212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:56:38.827225 17212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:56:38.831740 17212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:56:38.832044 17212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:56:38.833109 17212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:56:38.834465 17212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:56:38.834594 17212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:56:38.834724 17212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:56:38.834811 17212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:56:38.870919 17212 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46295
I20260620 02:56:38.871016 17339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46295 every 8 connection(s)
I20260620 02:56:38.872407 17212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:56:38.877223 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:38.875387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.2:39079)
0620 02:56:38.876135 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:38.876357 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":278,"threads_started":1}
I20260620 02:56:38.877214 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 17212
W20260620 02:56:38.877919 17340 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34103 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:34103: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
I20260620 02:56:38.878027 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:45263
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:34103
--webserver_port=39989
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:56:38.878901 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:38.878538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.2:45629)
0620 02:56:38.878690 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:38.878786 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 02:56:38.879897 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:38.879433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.2:52807)
0620 02:56:38.879626 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:38.879732 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 02:56:38.880214 17340 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 02:56:39.218823 17343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:56:39.219161 17343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:56:39.219358 17343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:56:39.226603 17343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:56:39.226754 17343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:56:39.226933 17343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:56:39.236748 17343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:45263
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=39989
--tserver_master_addrs=127.0.97.62:34103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:56:39.238077 17343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:56:39.239441 17343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:56:39.253070 17348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:56:39.253208 17351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:56:39.253244 17349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:56:39.254369 17343 server_base.cc:1034] running on GCE node
I20260620 02:56:39.255095 17343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:56:39.256283 17343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:56:39.257598 17343 hybrid_clock.cc:648] HybridClock initialized: now 1781924199257568 us; error 56 us; skew 500 ppm
I20260620 02:56:39.257992 17343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:56:39 dist-test-slave-8p8c krb5kdc[15891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:56:39.261533 17343 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:56:39.263199 17343 webserver.cc:460] Webserver started at http://127.0.97.3:39989/ using document root <none> and password file <none>
I20260620 02:56:39.263911 17343 fs_manager.cc:359] Metadata directory not provided
I20260620 02:56:39.264063 17343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:56:39.269510 17343 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 02:56:39.272337 17358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:56:39.273736 17343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:56:39.273913 17343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/wal
uuid: "85367245efeb4a08b1599f5ffcced30c"
format_stamp: "Formatted at 2026-06-20 02:55:53 on dist-test-slave-8p8c"
server_key: "9717a6c83bf6296baa55c03fe57e9bd1"
server_key_iv: "676b9c17c40aa29b562bf1f6073e92b4"
server_key_version: "encryptionkey@0"
I20260620 02:56:39.275013 17343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:56:39.294989 17343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:56:39.299420 17343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:56:39.299705 17343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:56:39.300721 17343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:56:39.302060 17343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:56:39.302205 17343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:56:39.302343 17343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:56:39.302487 17343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 02:56:39.337178 17343 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45263
I20260620 02:56:39.337267 17470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45263 every 8 connection(s)
I20260620 02:56:39.338774 17343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:56:39.343636 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:39.341793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.3:34693)
0620 02:56:39.342556 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:39.342768 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":272,"threads_started":1}
W20260620 02:56:39.344305 17471 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34103 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:34103: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
I20260620 02:56:39.344869 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 17343
I20260620 02:56:39.345170 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:39.344823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.3:51633)
0620 02:56:39.344966 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:39.345049 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 02:56:39.345983 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:39.345665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.3:39857)
0620 02:56:39.345799 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:39.345879 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 02:56:39.346177 17471 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 02:56:39.397166 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:39.396395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.1:57025)
0620 02:56:39.396838 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:39.396925 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":232,"threads_started":1}
I20260620 02:56:39.881762 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:39.880903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.2:48975)
0620 02:56:39.881385 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:39.881515 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":235,"threads_started":1}
I20260620 02:56:40.347855 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:40.346891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.3:50225)
0620 02:56:40.347493 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:40.347583 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":337,"threads_started":1}
I20260620 02:56:40.398792 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:40.398073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.1:49047)
0620 02:56:40.398547 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:40.398635 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":225,"threads_started":1}
I20260620 02:56:40.883527 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:40.882789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.2:37783)
0620 02:56:40.883274 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:40.883360 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":239,"threads_started":1}
I20260620 02:56:41.349728 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:41.348917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.3:56497)
0620 02:56:41.349443 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:41.349547 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":258,"threads_started":1}
I20260620 02:56:41.400751 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:41.399866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.1:54361)
0620 02:56:41.400455 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:41.400568 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":316,"threads_started":1}
I20260620 02:56:41.885390 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:41.884596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.2:59943)
0620 02:56:41.885129 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:41.885214 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":260,"threads_started":1}
I20260620 02:56:42.351621 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:42.350888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.3:43341)
0620 02:56:42.351348 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:42.351444 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":238,"threads_started":1}
I20260620 02:56:42.402498 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:42.401660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.1:34283)
0620 02:56:42.402164 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:42.402246 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":237,"threads_started":1}
I20260620 02:56:42.887434 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:42.886602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.2:59999)
0620 02:56:42.887148 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:42.887250 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":248,"threads_started":1}
I20260620 02:56:43.353523 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:43.352674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.3:35643)
0620 02:56:43.353235 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:43.353324 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":308,"threads_started":1}
I20260620 02:56:43.404343 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:43.403465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.1:60479)
0620 02:56:43.404074 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:43.404170 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":312,"threads_started":1}
I20260620 02:56:43.889168 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:43.888336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.2:36561)
0620 02:56:43.888848 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:43.888981 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":261,"threads_started":1}
I20260620 02:56:44.355417 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:44.354461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.3:52587)
0620 02:56:44.355109 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:44.355200 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":393,"threads_started":1}
I20260620 02:56:44.406255 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:44.405533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.1:60135)
0620 02:56:44.406007 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:44.406093 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":243,"threads_started":1}
I20260620 02:56:44.891244 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:44.890279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.2:36781)
0620 02:56:44.890916 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:44.891041 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":364,"threads_started":1}
I20260620 02:56:45.357336 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:45.356490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.3:56931)
0620 02:56:45.357036 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:45.357148 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":259,"threads_started":1}
I20260620 02:56:45.408321 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:45.407259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.1:56403)
0620 02:56:45.407979 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:45.408079 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":412,"threads_started":1}
I20260620 02:56:45.893373 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:45.892448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.2:55659)
0620 02:56:45.893023 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:45.893140 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":268,"threads_started":1}
I20260620 02:56:46.359328 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:46.358447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.3:41935)
0620 02:56:46.359006 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:46.359095 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":256,"threads_started":1}
I20260620 02:56:46.410243 17496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:46.409422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.1:57615)
0620 02:56:46.409975 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:46.410064 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":244,"threads_started":1}
I20260620 02:56:46.895228 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:46.894506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.2:54979)
0620 02:56:46.894972 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:46.895057 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":239,"threads_started":1}
I20260620 02:56:47.361100 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:47.360388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.3:40653)
0620 02:56:47.360857 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:47.360942 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":249,"threads_started":1}
I20260620 02:56:47.411967 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:47.411149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.1:33043)
0620 02:56:47.411645 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:47.411743 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":247,"threads_started":1}
I20260620 02:56:47.897313 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:47.896443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.2:41705)
0620 02:56:47.897009 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:47.897132 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":291,"threads_started":1}
I20260620 02:56:48.363236 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:48.362318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.3:36297)
0620 02:56:48.362950 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:48.363055 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":369,"threads_started":1}
I20260620 02:56:48.414116 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:48.413120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.1:59671)
0620 02:56:48.413734 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:48.413853 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":342,"threads_started":1}
I20260620 02:56:48.899297 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:48.898577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.2:35779)
0620 02:56:48.899043 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:48.899131 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":242,"threads_started":1}
I20260620 02:56:49.365172 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:49.364382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.3:55887)
0620 02:56:49.364924 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:49.365000 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":330,"threads_started":1}
I20260620 02:56:49.415903 17505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:49.415052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.1:55185)
0620 02:56:49.415639 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:49.415721 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":295,"threads_started":1}
I20260620 02:56:49.901296 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:49.900232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.2:40019)
0620 02:56:49.900967 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:49.901115 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":493,"threads_started":1}
I20260620 02:56:50.367218 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:50.366274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.3:34363)
0620 02:56:50.366887 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:50.366992 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":365,"threads_started":1}
I20260620 02:56:50.417482 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:50.416815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.1:58361)
0620 02:56:50.417248 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:50.417327 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":238,"threads_started":1}
I20260620 02:56:50.903251 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:50.902338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.2:34193)
0620 02:56:50.902976 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:50.903080 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":330,"threads_started":1}
I20260620 02:56:51.369015 17510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:51.368277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.3:42863)
0620 02:56:51.368759 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:51.368844 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":266,"threads_started":1}
I20260620 02:56:51.419337 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:51.418542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.1:58037)
0620 02:56:51.419017 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:51.419097 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":250,"threads_started":1}
I20260620 02:56:51.904912 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:51.904125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.2:50777)
0620 02:56:51.904640 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:51.904725 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":276,"threads_started":1}
I20260620 02:56:52.371104 17513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:52.370058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.3:43951)
0620 02:56:52.370651 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:52.370745 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":260,"threads_started":1}
I20260620 02:56:52.421044 17514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:52.420281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.1:40843)
0620 02:56:52.420746 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:52.420829 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":245,"threads_started":1}
I20260620 02:56:52.906587 17515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:52.905785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.2:33705)
0620 02:56:52.906225 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:52.906308 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":229,"threads_started":1}
I20260620 02:56:53.373322 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:53.372247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.3:35057)
0620 02:56:53.372858 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:53.372985 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":296,"threads_started":1}
I20260620 02:56:53.423130 17517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:53.422231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.1:60373)
0620 02:56:53.422799 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:53.422890 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":327,"threads_started":1}
I20260620 02:56:53.908624 17518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:53.907841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.2:54239)
0620 02:56:53.908335 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:53.908434 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":274,"threads_started":1}
I20260620 02:56:54.348393 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 17040
I20260620 02:56:54.368641 17519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:54.349651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44100 (local address 127.0.97.1:40417)
0620 02:56:54.350343 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0620 02:56:54.350389 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0620 02:56:54.350443 (+ 54us) server_negotiation.cc:407] Connection header received
0620 02:56:54.352453 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:56:54.352464 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:56:54.352920 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:56:54.353237 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:56:54.354802 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:54.356281 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:54.357780 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:54.358596 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:54.358749 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:54.361584 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:56:54.361592 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:56:54.361609 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:56:54.361649 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:56:54.364873 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:54.365564 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:54.365571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:54.365578 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:54.365749 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:54.366154 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:54.366158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:54.366161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:54.366875 (+ 714us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:56:54.367171 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 02:56:54.367777 (+ 606us) server_negotiation.cc:299] Negotiation successful
0620 02:56:54.368315 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":318,"threads_started":1}
I20260620 02:56:54.370153 17204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:56:54.375478 17521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:54.374640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34103 (local address 127.0.97.3:41881)
0620 02:56:54.375143 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:54.375258 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":269,"threads_started":1}
I20260620 02:56:54.460649 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 17040
I20260620 02:56:54.478407 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 17212
I20260620 02:56:54.498358 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:54.479498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42846 (local address 127.0.97.2:46295)
0620 02:56:54.480218 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0620 02:56:54.480233 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:56:54.480279 (+ 46us) server_negotiation.cc:407] Connection header received
0620 02:56:54.481996 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:56:54.482005 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:56:54.482546 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:56:54.482860 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:56:54.484269 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:54.485581 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:54.487052 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:54.487813 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:54.487974 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:54.490878 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:56:54.490886 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:56:54.490904 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:56:54.490948 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:56:54.494658 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:54.495517 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:54.495527 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:54.495537 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:54.495725 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:54.496197 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:54.496201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:54.496204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:54.496806 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:56:54.497181 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0620 02:56:54.497623 (+ 442us) server_negotiation.cc:299] Negotiation successful
0620 02:56:54.497991 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":341,"threads_started":1}
I20260620 02:56:54.499995 17335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:56:54.583684 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 17212
I20260620 02:56:54.602298 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 17343
I20260620 02:56:54.621147 17525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:54.603346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52684 (local address 127.0.97.3:45263)
0620 02:56:54.603984 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0620 02:56:54.604000 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 02:56:54.604043 (+ 43us) server_negotiation.cc:407] Connection header received
0620 02:56:54.605658 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:56:54.605668 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:56:54.606118 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:56:54.606507 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:56:54.608011 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:54.609196 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:54.610427 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:54.611080 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:54.611274 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:54.614303 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:56:54.614311 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:56:54.614326 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:56:54.614498 (+ 172us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:56:54.617582 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:54.618442 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:54.618452 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:54.618467 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:54.618695 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:54.619057 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:54.619060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:54.619062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:54.619622 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:56:54.619848 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 02:56:54.620343 (+ 495us) server_negotiation.cc:299] Negotiation successful
0620 02:56:54.620826 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":292,"threads_started":1}
I20260620 02:56:54.622968 17466 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:56:54.710063 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 17343
2026-06-20T02:56:54Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64578 ms)
[----------] 30 tests from SecurityITest (300250 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:56:54 dist-test-slave-8p8c krb5kdc[17529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:56:54 dist-test-slave-8p8c krb5kdc[17529](info): set up 2 sockets
Jun 20 02:56:54 dist-test-slave-8p8c krb5kdc[17529](info): commencing operation
krb5kdc: starting...
W20260620 02:56:55.830677 388 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.008s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:56:56 dist-test-slave-8p8c krb5kdc[17529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:56:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:56:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:56:56.095378 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36523
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:56:56.431774 17542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:56:56.432250 17542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:56:56.432428 17542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:56:56.441015 17542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:56:56.441185 17542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:56:56.441303 17542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:56:56.441404 17542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:56:56.441494 17542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:56:56.452003 17542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:56:56.453408 17542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:56:56.454897 17542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:56:56.468688 17548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:56:56.468721 17547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:56:56.468837 17542 server_base.cc:1034] running on GCE node
W20260620 02:56:56.468801 17550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:56:56.470304 17542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:56:56.471922 17542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:56:56.473150 17542 hybrid_clock.cc:648] HybridClock initialized: now 1781924216473089 us; error 45 us; skew 500 ppm
I20260620 02:56:56.473544 17542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:56:56 dist-test-slave-8p8c krb5kdc[17529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:56:56.477322 17542 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:56:56.479005 17542 webserver.cc:460] Webserver started at http://127.0.97.62:46327/ using document root <none> and password file <none>
I20260620 02:56:56.479756 17542 fs_manager.cc:359] Metadata directory not provided
I20260620 02:56:56.479859 17542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:56:56.480201 17542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:56:56.483400 17542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "e38ca8d711ee4066af3a2d4485a6e70e"
format_stamp: "Formatted at 2026-06-20 02:56:56 on dist-test-slave-8p8c"
server_key: "d2304810b8a5e5c823365493698141a4"
server_key_iv: "5f597a1ba33249f154dcbe37a60ccd5e"
server_key_version: "encryptionkey@0"
I20260620 02:56:56.484205 17542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "e38ca8d711ee4066af3a2d4485a6e70e"
format_stamp: "Formatted at 2026-06-20 02:56:56 on dist-test-slave-8p8c"
server_key: "d2304810b8a5e5c823365493698141a4"
server_key_iv: "5f597a1ba33249f154dcbe37a60ccd5e"
server_key_version: "encryptionkey@0"
I20260620 02:56:56.489777 17542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:56:56.493955 17557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:56:56.495785 17542 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260620 02:56:56.495985 17542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "e38ca8d711ee4066af3a2d4485a6e70e"
format_stamp: "Formatted at 2026-06-20 02:56:56 on dist-test-slave-8p8c"
server_key: "d2304810b8a5e5c823365493698141a4"
server_key_iv: "5f597a1ba33249f154dcbe37a60ccd5e"
server_key_version: "encryptionkey@0"
I20260620 02:56:56.496212 17542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:56:56.520955 17542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:56:56.526042 17542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:56:56.526474 17542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:56:56.548126 17542 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36523
I20260620 02:56:56.548216 17608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36523 every 8 connection(s)
I20260620 02:56:56.549654 17542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:56:56.550069 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 17542
I20260620 02:56:56.550289 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:56:56.551149 388 external_mini_cluster.cc:1428] Setting key f81a623a928fcfe2091c7eb943ab6b8e
I20260620 02:56:56.555639 17609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:56:56.569176 17609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e38ca8d711ee4066af3a2d4485a6e70e: Bootstrap starting.
Jun 20 02:56:56 dist-test-slave-8p8c krb5kdc[17529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:56:56.574041 17609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e38ca8d711ee4066af3a2d4485a6e70e: Neither blocks nor log segments found. Creating new log.
I20260620 02:56:56.575680 17609 log.cc:824] T 00000000000000000000000000000000 P e38ca8d711ee4066af3a2d4485a6e70e: Log is configured to *not* fsync() on all Append() calls
I20260620 02:56:56.579495 17609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e38ca8d711ee4066af3a2d4485a6e70e: No bootstrap required, opened a new log
I20260620 02:56:56.581383 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:56.554421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55026 (local address 127.0.97.62:36523)
0620 02:56:56.555566 (+ 1145us) server_negotiation.cc:206] Beginning negotiation
0620 02:56:56.555584 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 02:56:56.556008 (+ 424us) server_negotiation.cc:407] Connection header received
0620 02:56:56.558546 (+ 2538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:56:56.558577 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:56:56.559164 (+ 587us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:56:56.559672 (+ 508us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:56:56.561079 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:56.563200 (+ 2121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:56.564959 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:56.566070 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:56.566281 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:56.571347 (+ 5066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:56:56.571364 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:56:56.571388 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:56:56.571459 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:56:56.575999 (+ 4540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:56.577182 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:56.577188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:56.577195 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:56.577452 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:56.577919 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:56.577925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:56.577928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:56.578717 (+ 789us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:56:56.579011 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 02:56:56.579456 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 02:56:56.579945 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":887,"thread_start_us":567,"threads_started":1}
I20260620 02:56:56.588251 17609 raft_consensus.cc:348] T 00000000000000000000000000000000 P e38ca8d711ee4066af3a2d4485a6e70e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e38ca8d711ee4066af3a2d4485a6e70e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36523 } }
I20260620 02:56:56.588613 17609 raft_consensus.cc:374] T 00000000000000000000000000000000 P e38ca8d711ee4066af3a2d4485a6e70e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:56:56.588711 17609 raft_consensus.cc:729] T 00000000000000000000000000000000 P e38ca8d711ee4066af3a2d4485a6e70e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e38ca8d711ee4066af3a2d4485a6e70e, State: Initialized, Role: FOLLOWER
I20260620 02:56:56.589275 17609 consensus_queue.cc:260] T 00000000000000000000000000000000 P e38ca8d711ee4066af3a2d4485a6e70e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e38ca8d711ee4066af3a2d4485a6e70e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36523 } }
I20260620 02:56:56.589594 17609 raft_consensus.cc:388] T 00000000000000000000000000000000 P e38ca8d711ee4066af3a2d4485a6e70e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:56:56.589769 17609 raft_consensus.cc:482] T 00000000000000000000000000000000 P e38ca8d711ee4066af3a2d4485a6e70e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:56:56.589962 17609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e38ca8d711ee4066af3a2d4485a6e70e [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:56:56.592196 17609 raft_consensus.cc:504] T 00000000000000000000000000000000 P e38ca8d711ee4066af3a2d4485a6e70e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e38ca8d711ee4066af3a2d4485a6e70e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36523 } }
I20260620 02:56:56.592707 17609 leader_election.cc:302] T 00000000000000000000000000000000 P e38ca8d711ee4066af3a2d4485a6e70e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e38ca8d711ee4066af3a2d4485a6e70e; no voters:
I20260620 02:56:56.593304 17609 leader_election.cc:288] T 00000000000000000000000000000000 P e38ca8d711ee4066af3a2d4485a6e70e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:56:56.593528 17614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e38ca8d711ee4066af3a2d4485a6e70e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:56:56.594261 17614 raft_consensus.cc:686] T 00000000000000000000000000000000 P e38ca8d711ee4066af3a2d4485a6e70e [term 1 LEADER]: Becoming Leader. State: Replica: e38ca8d711ee4066af3a2d4485a6e70e, State: Running, Role: LEADER
I20260620 02:56:56.595026 17614 consensus_queue.cc:237] T 00000000000000000000000000000000 P e38ca8d711ee4066af3a2d4485a6e70e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e38ca8d711ee4066af3a2d4485a6e70e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36523 } }
I20260620 02:56:56.595654 17609 sys_catalog.cc:564] T 00000000000000000000000000000000 P e38ca8d711ee4066af3a2d4485a6e70e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:56:56.598091 17615 sys_catalog.cc:455] T 00000000000000000000000000000000 P e38ca8d711ee4066af3a2d4485a6e70e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e38ca8d711ee4066af3a2d4485a6e70e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e38ca8d711ee4066af3a2d4485a6e70e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36523 } } }
I20260620 02:56:56.598594 17615 sys_catalog.cc:458] T 00000000000000000000000000000000 P e38ca8d711ee4066af3a2d4485a6e70e [sys.catalog]: This master's current role is: LEADER
I20260620 02:56:56.599210 17616 sys_catalog.cc:455] T 00000000000000000000000000000000 P e38ca8d711ee4066af3a2d4485a6e70e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e38ca8d711ee4066af3a2d4485a6e70e. Latest consensus state: current_term: 1 leader_uuid: "e38ca8d711ee4066af3a2d4485a6e70e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e38ca8d711ee4066af3a2d4485a6e70e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36523 } } }
I20260620 02:56:56.600961 17616 sys_catalog.cc:458] T 00000000000000000000000000000000 P e38ca8d711ee4066af3a2d4485a6e70e [sys.catalog]: This master's current role is: LEADER
I20260620 02:56:56.603185 17623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:56:56.608923 17623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:56:56.615557 17631 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 02:56:56.619356 17623 catalog_manager.cc:1347] Generated new cluster ID: 92aa66ca9d87405598716cf24b7e6cd8
I20260620 02:56:56.619478 17623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:56:56.671171 17623 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:56:56.672530 17623 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:56:56.682399 17623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e38ca8d711ee4066af3a2d4485a6e70e: Generated new TSK 0
I20260620 02:56:56.683259 17623 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:56:56.756127 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36523
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:56:57.099565 17635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:56:57.099910 17635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:56:57.100080 17635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:56:57.107928 17635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:56:57.108079 17635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:56:57.108273 17635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:56:57.118165 17635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:56:57.119558 17635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:56:57.120903 17635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:56:57.134146 17643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:56:57.134209 17641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:56:57.134244 17640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:56:57.135195 17635 server_base.cc:1034] running on GCE node
I20260620 02:56:57.135795 17635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:56:57.136720 17635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:56:57.137923 17635 hybrid_clock.cc:648] HybridClock initialized: now 1781924217137869 us; error 44 us; skew 500 ppm
I20260620 02:56:57.138262 17635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:56:57 dist-test-slave-8p8c krb5kdc[17529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:56:57.141433 17635 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:56:57.142938 17635 webserver.cc:460] Webserver started at http://127.0.97.1:43411/ using document root <none> and password file <none>
I20260620 02:56:57.143599 17635 fs_manager.cc:359] Metadata directory not provided
I20260620 02:56:57.143698 17635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:56:57.143944 17635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:56:57.146948 17635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-0/data/instance:
uuid: "18902428d4b9433c974d2509bbfe736f"
format_stamp: "Formatted at 2026-06-20 02:56:57 on dist-test-slave-8p8c"
server_key: "0f9df266d5067ce396907eb2c9d231d8"
server_key_iv: "ef8e22b2998ab72ba9bea2fbae1d672c"
server_key_version: "encryptionkey@0"
I20260620 02:56:57.147673 17635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-0/wal/instance:
uuid: "18902428d4b9433c974d2509bbfe736f"
format_stamp: "Formatted at 2026-06-20 02:56:57 on dist-test-slave-8p8c"
server_key: "0f9df266d5067ce396907eb2c9d231d8"
server_key_iv: "ef8e22b2998ab72ba9bea2fbae1d672c"
server_key_version: "encryptionkey@0"
I20260620 02:56:57.152894 17635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:56:57.156347 17650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:56:57.157836 17635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:56:57.158030 17635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-0/wal
uuid: "18902428d4b9433c974d2509bbfe736f"
format_stamp: "Formatted at 2026-06-20 02:56:57 on dist-test-slave-8p8c"
server_key: "0f9df266d5067ce396907eb2c9d231d8"
server_key_iv: "ef8e22b2998ab72ba9bea2fbae1d672c"
server_key_version: "encryptionkey@0"
I20260620 02:56:57.158255 17635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:56:57.179664 17635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:56:57.184928 17635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:56:57.185258 17635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:56:57.186318 17635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:56:57.187793 17635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:56:57.187896 17635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:56:57.188047 17635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:56:57.188117 17635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:56:57.223508 17635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36647
I20260620 02:56:57.223603 17762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36647 every 8 connection(s)
I20260620 02:56:57.224830 17635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:56:57.234745 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 17635
I20260620 02:56:57.234980 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:56:57.235633 388 external_mini_cluster.cc:1428] Setting key 25b7d84cff2c56c9bcba5498e3f81bf2
Jun 20 02:56:57 dist-test-slave-8p8c krb5kdc[17529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:56:57.245780 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:57.227686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48379 (local address 127.0.97.62:36523)
0620 02:56:57.228165 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0620 02:56:57.228170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:56:57.229217 (+ 1047us) server_negotiation.cc:407] Connection header received
0620 02:56:57.231208 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:56:57.231212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:56:57.231299 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:56:57.231416 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:56:57.232908 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:57.233655 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:57.235473 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:57.235983 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:57.236123 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:57.239708 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:56:57.239714 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:56:57.239725 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:56:57.239760 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:56:57.242703 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:57.243449 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:57.243455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:57.243459 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:57.243583 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:57.244046 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:57.244050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:57.244052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:57.244311 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:56:57.244468 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 02:56:57.245171 (+ 703us) server_negotiation.cc:299] Negotiation successful
0620 02:56:57.245397 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":238,"threads_started":1}
I20260620 02:56:57.246869 17766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:57.228299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36523 (local address 127.0.97.1:48379)
0620 02:56:57.229012 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:57.229051 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 02:56:57.230918 (+ 1867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:56:57.231657 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:56:57.231665 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:56:57.232184 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:56:57.232742 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:56:57.232759 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:56:57.233817 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:56:57.233821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:56:57.235328 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:56:57.235340 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:56:57.236219 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:56:57.236229 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:56:57.236484 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:57.237229 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:56:57.237270 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:56:57.239489 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:56:57.242847 (+ 3358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:56:57.242856 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:56:57.242874 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:56:57.243322 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:56:57.243707 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:56:57.243711 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:56:57.243715 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:56:57.243914 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:56:57.244440 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:56:57.244449 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:56:57.244799 (+ 350us) client_negotiation.cc:769] Sending connection context
0620 02:56:57.245065 (+ 266us) client_negotiation.cc:240] Negotiation successful
0620 02:56:57.245634 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":287,"threads_started":1}
I20260620 02:56:57.248611 17763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36523
I20260620 02:56:57.248944 17763 heartbeater.cc:464] Registering TS with master...
I20260620 02:56:57.249796 17763 heartbeater.cc:511] Master 127.0.97.62:36523 requested a full tablet report, sending...
I20260620 02:56:57.252279 17574 ts_manager.cc:194] Registered new tserver with Master: 18902428d4b9433c974d2509bbfe736f (127.0.97.1:36647)
I20260620 02:56:57.254484 17574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48379
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:56:57.295212 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36523
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:56:57.634649 17769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:56:57.634943 17769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:56:57.635147 17769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:56:57.642732 17769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:56:57.642845 17769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:56:57.643035 17769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:56:57.653311 17769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:56:57.654698 17769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:56:57.656198 17769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:56:57.670055 17774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:56:57.670113 17777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:56:57.670123 17775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:56:57.670837 17769 server_base.cc:1034] running on GCE node
I20260620 02:56:57.671422 17769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:56:57.672465 17769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:56:57.673707 17769 hybrid_clock.cc:648] HybridClock initialized: now 1781924217673656 us; error 45 us; skew 500 ppm
I20260620 02:56:57.674026 17769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:56:57 dist-test-slave-8p8c krb5kdc[17529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:56:57.677511 17769 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:56:57.679131 17769 webserver.cc:460] Webserver started at http://127.0.97.2:33743/ using document root <none> and password file <none>
I20260620 02:56:57.679807 17769 fs_manager.cc:359] Metadata directory not provided
I20260620 02:56:57.679913 17769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:56:57.680172 17769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:56:57.683233 17769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-1/data/instance:
uuid: "ee2dd6a4d5df4d93aa49baf1c98faa39"
format_stamp: "Formatted at 2026-06-20 02:56:57 on dist-test-slave-8p8c"
server_key: "1205c28ad8ab97a0ec4ea65caa3c29ff"
server_key_iv: "510b9d04ecd382b94ebc9c6d211b8171"
server_key_version: "encryptionkey@0"
I20260620 02:56:57.683980 17769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ee2dd6a4d5df4d93aa49baf1c98faa39"
format_stamp: "Formatted at 2026-06-20 02:56:57 on dist-test-slave-8p8c"
server_key: "1205c28ad8ab97a0ec4ea65caa3c29ff"
server_key_iv: "510b9d04ecd382b94ebc9c6d211b8171"
server_key_version: "encryptionkey@0"
I20260620 02:56:57.689302 17769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:56:57.693085 17784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:56:57.694728 17769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 02:56:57.694919 17769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-1/wal
uuid: "ee2dd6a4d5df4d93aa49baf1c98faa39"
format_stamp: "Formatted at 2026-06-20 02:56:57 on dist-test-slave-8p8c"
server_key: "1205c28ad8ab97a0ec4ea65caa3c29ff"
server_key_iv: "510b9d04ecd382b94ebc9c6d211b8171"
server_key_version: "encryptionkey@0"
I20260620 02:56:57.695135 17769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:56:57.712836 17769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:56:57.717483 17769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:56:57.717814 17769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:56:57.718974 17769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:56:57.720374 17769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:56:57.720479 17769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:56:57.720638 17769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:56:57.720726 17769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:56:57.757682 17769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44333
I20260620 02:56:57.757730 17896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44333 every 8 connection(s)
I20260620 02:56:57.759145 17769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:56:57.760452 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 17769
I20260620 02:56:57.760658 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:56:57.761205 388 external_mini_cluster.cc:1428] Setting key 382fe8a0f281bd8ac6648c76801603d5
Jun 20 02:56:57 dist-test-slave-8p8c krb5kdc[17529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:56:57.781651 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:57.762178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39021 (local address 127.0.97.62:36523)
0620 02:56:57.762788 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0620 02:56:57.762794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:56:57.764254 (+ 1460us) server_negotiation.cc:407] Connection header received
0620 02:56:57.766202 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:56:57.766210 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:56:57.766304 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:56:57.766452 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:56:57.768138 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:57.769324 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:57.771037 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:57.771493 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:57.771595 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:57.775777 (+ 4182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:56:57.775783 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:56:57.775786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:56:57.775822 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:56:57.778528 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:57.779267 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:57.779273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:57.779277 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:57.779411 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:57.779854 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:57.779859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:57.779861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:57.780127 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:56:57.780282 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 02:56:57.780987 (+ 705us) server_negotiation.cc:299] Negotiation successful
0620 02:56:57.781214 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":347,"threads_started":1}
I20260620 02:56:57.782884 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:57.763205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36523 (local address 127.0.97.2:39021)
0620 02:56:57.764072 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:57.764121 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 02:56:57.765902 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:56:57.766769 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:56:57.766778 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:56:57.767329 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:56:57.767940 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:56:57.767959 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:56:57.769494 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:56:57.769498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:56:57.770882 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:56:57.770896 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:56:57.771693 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:56:57.771698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:56:57.771891 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:57.772616 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:56:57.772643 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:56:57.775508 (+ 2865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:56:57.778673 (+ 3165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:56:57.778683 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:56:57.778700 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:56:57.779134 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:56:57.779532 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:56:57.779537 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:56:57.779541 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:56:57.779727 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:56:57.780253 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:56:57.780261 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:56:57.780620 (+ 359us) client_negotiation.cc:769] Sending connection context
0620 02:56:57.780893 (+ 273us) client_negotiation.cc:240] Negotiation successful
0620 02:56:57.781450 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":366,"threads_started":1}
I20260620 02:56:57.784463 17897 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36523
I20260620 02:56:57.784788 17897 heartbeater.cc:464] Registering TS with master...
I20260620 02:56:57.785503 17897 heartbeater.cc:511] Master 127.0.97.62:36523 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:56:57.787067 17574 ts_manager.cc:194] Registered new tserver with Master: ee2dd6a4d5df4d93aa49baf1c98faa39 (127.0.97.2:44333)
I20260620 02:56:57.787921 17574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:56:57.818802 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36523
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:56:58.149931 17903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:56:58.150233 17903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:56:58.150328 17903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:56:58.157608 17903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:56:58.157725 17903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:56:58.157838 17903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:56:58.168053 17903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:56:58.169340 17903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:56:58.170802 17903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:56:58.184664 17908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:56:58.184819 17911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:56:58.184871 17909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:56:58.186206 17903 server_base.cc:1034] running on GCE node
I20260620 02:56:58.186980 17903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:56:58.187960 17903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:56:58.189189 17903 hybrid_clock.cc:648] HybridClock initialized: now 1781924218189120 us; error 57 us; skew 500 ppm
I20260620 02:56:58.189584 17903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:56:58 dist-test-slave-8p8c krb5kdc[17529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:56:58.197003 17903 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:56:58.198917 17903 webserver.cc:460] Webserver started at http://127.0.97.3:38793/ using document root <none> and password file <none>
I20260620 02:56:58.199735 17903 fs_manager.cc:359] Metadata directory not provided
I20260620 02:56:58.199869 17903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:56:58.200284 17903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:56:58.203786 17903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-2/data/instance:
uuid: "dc1d815d1e3448eea62052e286a1aca2"
format_stamp: "Formatted at 2026-06-20 02:56:58 on dist-test-slave-8p8c"
server_key: "c71a27d59e627e4f78c41fcd7ee0566e"
server_key_iv: "069161484f3005f32ff86217ff643cef"
server_key_version: "encryptionkey@0"
I20260620 02:56:58.204679 17903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-2/wal/instance:
uuid: "dc1d815d1e3448eea62052e286a1aca2"
format_stamp: "Formatted at 2026-06-20 02:56:58 on dist-test-slave-8p8c"
server_key: "c71a27d59e627e4f78c41fcd7ee0566e"
server_key_iv: "069161484f3005f32ff86217ff643cef"
server_key_version: "encryptionkey@0"
I20260620 02:56:58.209988 17903 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:56:58.213747 17918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:56:58.215323 17903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:56:58.215618 17903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-2/wal
uuid: "dc1d815d1e3448eea62052e286a1aca2"
format_stamp: "Formatted at 2026-06-20 02:56:58 on dist-test-slave-8p8c"
server_key: "c71a27d59e627e4f78c41fcd7ee0566e"
server_key_iv: "069161484f3005f32ff86217ff643cef"
server_key_version: "encryptionkey@0"
I20260620 02:56:58.215940 17903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:56:58.240265 17903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:56:58.244982 17903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:56:58.245298 17903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:56:58.246320 17903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:56:58.247763 17903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:56:58.247838 17903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:56:58.247921 17903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:56:58.247963 17903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:56:58.258845 17763 heartbeater.cc:503] Master 127.0.97.62:36523 was elected leader, sending a full tablet report...
I20260620 02:56:58.283269 17903 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46873
I20260620 02:56:58.283352 18030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46873 every 8 connection(s)
I20260620 02:56:58.284583 17903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:56:58.294216 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 17903
I20260620 02:56:58.294428 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:56:58.294888 388 external_mini_cluster.cc:1428] Setting key ed300dffb448546552ee35e754ca7c44
Jun 20 02:56:58 dist-test-slave-8p8c krb5kdc[17529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:56:58.307399 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:58.287427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49813 (local address 127.0.97.62:36523)
0620 02:56:58.287930 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0620 02:56:58.287937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:56:58.288904 (+ 967us) server_negotiation.cc:407] Connection header received
0620 02:56:58.290861 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:56:58.290865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:56:58.290936 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:56:58.291201 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:56:58.292712 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.293505 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.295552 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.296224 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.296360 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:58.301140 (+ 4780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:56:58.301147 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:56:58.301151 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:56:58.301190 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:56:58.304196 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:58.304944 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:58.304949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:58.304954 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:58.305075 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:58.305534 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:58.305538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:58.305540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:58.305815 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:56:58.305972 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 02:56:58.306773 (+ 801us) server_negotiation.cc:299] Negotiation successful
0620 02:56:58.307006 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":212,"threads_started":1}
I20260620 02:56:58.308377 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:58.287968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36523 (local address 127.0.97.3:49813)
0620 02:56:58.288734 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:58.288778 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 02:56:58.290566 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:56:58.291436 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:56:58.291444 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:56:58.291953 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:56:58.292529 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:56:58.292545 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.293667 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.293671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:56:58.295347 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:56:58.295386 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.296550 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.296558 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:56:58.296895 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:58.297980 (+ 1085us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:56:58.298004 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:56:58.300871 (+ 2867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:56:58.304349 (+ 3478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:56:58.304359 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:56:58.304383 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:56:58.304815 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:56:58.305194 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:56:58.305199 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:56:58.305203 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:56:58.305408 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:56:58.305947 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:56:58.305955 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:56:58.306315 (+ 360us) client_negotiation.cc:769] Sending connection context
0620 02:56:58.306652 (+ 337us) client_negotiation.cc:240] Negotiation successful
0620 02:56:58.307250 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":313,"threads_started":1}
I20260620 02:56:58.310010 18031 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36523
I20260620 02:56:58.310442 18031 heartbeater.cc:464] Registering TS with master...
I20260620 02:56:58.311305 18031 heartbeater.cc:511] Master 127.0.97.62:36523 requested a full tablet report, sending...
I20260620 02:56:58.312855 17574 ts_manager.cc:194] Registered new tserver with Master: dc1d815d1e3448eea62052e286a1aca2 (127.0.97.3:46873)
I20260620 02:56:58.313805 17574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49813
I20260620 02:56:58.323791 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:56:58.341961 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:58.328541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55034 (local address 127.0.97.62:36523)
0620 02:56:58.328882 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0620 02:56:58.328889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:56:58.329221 (+ 332us) server_negotiation.cc:407] Connection header received
0620 02:56:58.329347 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:56:58.329350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:56:58.329417 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:56:58.329507 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:56:58.331150 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.331891 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.333248 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.333685 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.333801 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:58.336143 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:56:58.336148 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:56:58.336151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:56:58.336184 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:56:58.339179 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:58.339807 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:58.339810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:58.339813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:58.339929 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:58.340477 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:58.340481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:58.340483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:58.340809 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:56:58.341004 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 02:56:58.341378 (+ 374us) server_negotiation.cc:299] Negotiation successful
0620 02:56:58.341584 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260620 02:56:58.345983 17574 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:56:58.352887 17574 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:56:58.377394 18044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:58.365516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46873 (local address 127.0.0.1:51452)
0620 02:56:58.366342 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:58.366465 (+ 123us) client_negotiation.cc:174] Beginning negotiation
0620 02:56:58.366701 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:56:58.370433 (+ 3732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:56:58.370438 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:56:58.370463 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:56:58.371002 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:56:58.371017 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.372383 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.372392 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:56:58.374843 (+ 2451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:56:58.374854 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.376481 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.376485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:56:58.376623 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:58.376662 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 02:56:58.376757 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 02:56:58.376879 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":204,"thread_start_us":1313,"threads_started":1}
I20260620 02:56:58.378095 18045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:58.365601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36647 (local address 127.0.0.1:48420)
0620 02:56:58.367334 (+ 1733us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:58.367352 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:56:58.367458 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:56:58.369592 (+ 2134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:56:58.369599 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:56:58.369631 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:56:58.370033 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:56:58.370047 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.372329 (+ 2282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.372337 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:56:58.375269 (+ 2932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:56:58.375283 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.377111 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.377117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:56:58.377313 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:58.377358 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 02:56:58.377456 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 02:56:58.377697 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1606,"mutex_wait_us":150,"thread_start_us":2006,"threads_started":1}
I20260620 02:56:58.378249 18047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:58.366082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51452 (local address 127.0.97.3:46873)
0620 02:56:58.369798 (+ 3716us) server_negotiation.cc:206] Beginning negotiation
0620 02:56:58.369808 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:56:58.369843 (+ 35us) server_negotiation.cc:407] Connection header received
0620 02:56:58.369936 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:56:58.369943 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:56:58.370179 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:56:58.370761 (+ 582us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:56:58.371181 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.372194 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.375000 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.376305 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.376437 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:58.376674 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 02:56:58.376838 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 02:56:58.377027 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3535,"thread_start_us":3362,"threads_started":1}
I20260620 02:56:58.379618 18046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:58.365976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48420 (local address 127.0.97.1:36647)
0620 02:56:58.368916 (+ 2940us) server_negotiation.cc:206] Beginning negotiation
0620 02:56:58.368926 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:56:58.368962 (+ 36us) server_negotiation.cc:407] Connection header received
0620 02:56:58.369082 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:56:58.369092 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:56:58.369362 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:56:58.370241 (+ 879us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:56:58.370516 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.372184 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.375433 (+ 3249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.376858 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.376983 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:58.377194 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 02:56:58.379156 (+ 1962us) server_negotiation.cc:299] Negotiation successful
0620 02:56:58.379275 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2782,"thread_start_us":2636,"threads_started":1}
I20260620 02:56:58.380162 18048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:58.367346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44333 (local address 127.0.0.1:43916)
0620 02:56:58.368199 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:58.368219 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:56:58.368350 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:56:58.370228 (+ 1878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:56:58.370232 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:56:58.370259 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:56:58.370808 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:56:58.370824 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.372785 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.372792 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:56:58.375272 (+ 2480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:56:58.375283 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.379417 (+ 4134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.379435 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:56:58.379568 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:58.379591 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 02:56:58.379695 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 02:56:58.379814 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":68,"thread_start_us":785,"threads_started":1}
I20260620 02:56:58.380237 18049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:58.367916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43916 (local address 127.0.97.2:44333)
0620 02:56:58.369300 (+ 1384us) server_negotiation.cc:206] Beginning negotiation
0620 02:56:58.369308 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:56:58.369334 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:56:58.369431 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:56:58.369437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:56:58.370027 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:56:58.371044 (+ 1017us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:56:58.371243 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.372655 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.377281 (+ 4626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.378979 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.379109 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:58.379282 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 02:56:58.379742 (+ 460us) server_negotiation.cc:299] Negotiation successful
0620 02:56:58.379890 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1248,"thread_start_us":1121,"threads_started":1}
I20260620 02:56:58.381834 17966 tablet_service.cc:1467] Processing CreateTablet for tablet 9248cd6e1cb942fba8a12af94ac6fc44 (DEFAULT_TABLE table=test-table [id=ef1f3f121cf14fc686e68c90b9060cf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:56:58.383411 17966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9248cd6e1cb942fba8a12af94ac6fc44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:56:58.383356 17832 tablet_service.cc:1467] Processing CreateTablet for tablet 9248cd6e1cb942fba8a12af94ac6fc44 (DEFAULT_TABLE table=test-table [id=ef1f3f121cf14fc686e68c90b9060cf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:56:58.384738 17832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9248cd6e1cb942fba8a12af94ac6fc44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:56:58.385742 17698 tablet_service.cc:1467] Processing CreateTablet for tablet 9248cd6e1cb942fba8a12af94ac6fc44 (DEFAULT_TABLE table=test-table [id=ef1f3f121cf14fc686e68c90b9060cf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:56:58.387348 17698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9248cd6e1cb942fba8a12af94ac6fc44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:56:58.394564 18050 tablet_bootstrap.cc:492] T 9248cd6e1cb942fba8a12af94ac6fc44 P dc1d815d1e3448eea62052e286a1aca2: Bootstrap starting.
I20260620 02:56:58.398407 18050 tablet_bootstrap.cc:654] T 9248cd6e1cb942fba8a12af94ac6fc44 P dc1d815d1e3448eea62052e286a1aca2: Neither blocks nor log segments found. Creating new log.
I20260620 02:56:58.399857 18050 log.cc:824] T 9248cd6e1cb942fba8a12af94ac6fc44 P dc1d815d1e3448eea62052e286a1aca2: Log is configured to *not* fsync() on all Append() calls
I20260620 02:56:58.399853 18051 tablet_bootstrap.cc:492] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39: Bootstrap starting.
I20260620 02:56:58.403244 18050 tablet_bootstrap.cc:492] T 9248cd6e1cb942fba8a12af94ac6fc44 P dc1d815d1e3448eea62052e286a1aca2: No bootstrap required, opened a new log
I20260620 02:56:58.403537 18050 ts_tablet_manager.cc:1397] T 9248cd6e1cb942fba8a12af94ac6fc44 P dc1d815d1e3448eea62052e286a1aca2: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260620 02:56:58.403527 18051 tablet_bootstrap.cc:654] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39: Neither blocks nor log segments found. Creating new log.
I20260620 02:56:58.404218 18053 tablet_bootstrap.cc:492] T 9248cd6e1cb942fba8a12af94ac6fc44 P 18902428d4b9433c974d2509bbfe736f: Bootstrap starting.
I20260620 02:56:58.404772 18051 log.cc:824] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39: Log is configured to *not* fsync() on all Append() calls
I20260620 02:56:58.407747 18053 tablet_bootstrap.cc:654] T 9248cd6e1cb942fba8a12af94ac6fc44 P 18902428d4b9433c974d2509bbfe736f: Neither blocks nor log segments found. Creating new log.
I20260620 02:56:58.408751 18053 log.cc:824] T 9248cd6e1cb942fba8a12af94ac6fc44 P 18902428d4b9433c974d2509bbfe736f: Log is configured to *not* fsync() on all Append() calls
I20260620 02:56:58.411700 18053 tablet_bootstrap.cc:492] T 9248cd6e1cb942fba8a12af94ac6fc44 P 18902428d4b9433c974d2509bbfe736f: No bootstrap required, opened a new log
I20260620 02:56:58.411963 18053 ts_tablet_manager.cc:1397] T 9248cd6e1cb942fba8a12af94ac6fc44 P 18902428d4b9433c974d2509bbfe736f: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260620 02:56:58.412961 18051 tablet_bootstrap.cc:492] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39: No bootstrap required, opened a new log
I20260620 02:56:58.413237 18051 ts_tablet_manager.cc:1397] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260620 02:56:58.415459 18050 raft_consensus.cc:348] T 9248cd6e1cb942fba8a12af94ac6fc44 P dc1d815d1e3448eea62052e286a1aca2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1d815d1e3448eea62052e286a1aca2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46873 } } peers { permanent_uuid: "18902428d4b9433c974d2509bbfe736f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36647 } } peers { permanent_uuid: "ee2dd6a4d5df4d93aa49baf1c98faa39" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44333 } }
I20260620 02:56:58.415925 18050 raft_consensus.cc:374] T 9248cd6e1cb942fba8a12af94ac6fc44 P dc1d815d1e3448eea62052e286a1aca2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:56:58.416008 18050 raft_consensus.cc:729] T 9248cd6e1cb942fba8a12af94ac6fc44 P dc1d815d1e3448eea62052e286a1aca2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc1d815d1e3448eea62052e286a1aca2, State: Initialized, Role: FOLLOWER
I20260620 02:56:58.416704 18050 consensus_queue.cc:260] T 9248cd6e1cb942fba8a12af94ac6fc44 P dc1d815d1e3448eea62052e286a1aca2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1d815d1e3448eea62052e286a1aca2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46873 } } peers { permanent_uuid: "18902428d4b9433c974d2509bbfe736f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36647 } } peers { permanent_uuid: "ee2dd6a4d5df4d93aa49baf1c98faa39" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44333 } }
I20260620 02:56:58.417831 18031 heartbeater.cc:503] Master 127.0.97.62:36523 was elected leader, sending a full tablet report...
I20260620 02:56:58.418550 18050 ts_tablet_manager.cc:1428] T 9248cd6e1cb942fba8a12af94ac6fc44 P dc1d815d1e3448eea62052e286a1aca2: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260620 02:56:58.420706 18053 raft_consensus.cc:348] T 9248cd6e1cb942fba8a12af94ac6fc44 P 18902428d4b9433c974d2509bbfe736f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1d815d1e3448eea62052e286a1aca2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46873 } } peers { permanent_uuid: "18902428d4b9433c974d2509bbfe736f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36647 } } peers { permanent_uuid: "ee2dd6a4d5df4d93aa49baf1c98faa39" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44333 } }
I20260620 02:56:58.421162 18053 raft_consensus.cc:374] T 9248cd6e1cb942fba8a12af94ac6fc44 P 18902428d4b9433c974d2509bbfe736f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:56:58.421270 18053 raft_consensus.cc:729] T 9248cd6e1cb942fba8a12af94ac6fc44 P 18902428d4b9433c974d2509bbfe736f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18902428d4b9433c974d2509bbfe736f, State: Initialized, Role: FOLLOWER
I20260620 02:56:58.421924 18053 consensus_queue.cc:260] T 9248cd6e1cb942fba8a12af94ac6fc44 P 18902428d4b9433c974d2509bbfe736f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1d815d1e3448eea62052e286a1aca2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46873 } } peers { permanent_uuid: "18902428d4b9433c974d2509bbfe736f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36647 } } peers { permanent_uuid: "ee2dd6a4d5df4d93aa49baf1c98faa39" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44333 } }
I20260620 02:56:58.422988 18051 raft_consensus.cc:348] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1d815d1e3448eea62052e286a1aca2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46873 } } peers { permanent_uuid: "18902428d4b9433c974d2509bbfe736f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36647 } } peers { permanent_uuid: "ee2dd6a4d5df4d93aa49baf1c98faa39" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44333 } }
I20260620 02:56:58.423429 18051 raft_consensus.cc:374] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:56:58.423538 18051 raft_consensus.cc:729] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee2dd6a4d5df4d93aa49baf1c98faa39, State: Initialized, Role: FOLLOWER
I20260620 02:56:58.424130 18051 consensus_queue.cc:260] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1d815d1e3448eea62052e286a1aca2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46873 } } peers { permanent_uuid: "18902428d4b9433c974d2509bbfe736f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36647 } } peers { permanent_uuid: "ee2dd6a4d5df4d93aa49baf1c98faa39" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44333 } }
I20260620 02:56:58.424989 18053 ts_tablet_manager.cc:1428] T 9248cd6e1cb942fba8a12af94ac6fc44 P 18902428d4b9433c974d2509bbfe736f: Time spent starting tablet: real 0.013s user 0.007s sys 0.006s
I20260620 02:56:58.426728 18051 ts_tablet_manager.cc:1428] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39: Time spent starting tablet: real 0.013s user 0.011s sys 0.004s
I20260620 02:56:58.428360 17897 heartbeater.cc:503] Master 127.0.97.62:36523 was elected leader, sending a full tablet report...
I20260620 02:56:58.458813 18058 raft_consensus.cc:482] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:56:58.459297 18058 raft_consensus.cc:504] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1d815d1e3448eea62052e286a1aca2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46873 } } peers { permanent_uuid: "18902428d4b9433c974d2509bbfe736f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36647 } } peers { permanent_uuid: "ee2dd6a4d5df4d93aa49baf1c98faa39" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44333 } }
I20260620 02:56:58.461335 18058 leader_election.cc:288] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc1d815d1e3448eea62052e286a1aca2 (127.0.97.3:46873), 18902428d4b9433c974d2509bbfe736f (127.0.97.1:36647)
I20260620 02:56:58.469738 18047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:58.461884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41645 (local address 127.0.97.3:46873)
0620 02:56:58.462129 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 02:56:58.462138 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:56:58.463193 (+ 1055us) server_negotiation.cc:407] Connection header received
0620 02:56:58.463461 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:56:58.463466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:56:58.463554 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:56:58.463672 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:56:58.464420 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.465484 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.467920 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.468700 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.468804 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:58.468942 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 02:56:58.469310 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 02:56:58.469450 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 02:56:58.469759 18046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:58.461808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47767 (local address 127.0.97.1:36647)
0620 02:56:58.462094 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0620 02:56:58.462103 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:56:58.462994 (+ 891us) server_negotiation.cc:407] Connection header received
0620 02:56:58.463211 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:56:58.463217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:56:58.463299 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:56:58.463386 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:56:58.464118 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.465098 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.467972 (+ 2874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.468736 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.468914 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:58.469014 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 02:56:58.469376 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 02:56:58.469511 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 02:56:58.469883 18059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:58.461743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36647 (local address 127.0.97.2:47767)
0620 02:56:58.462864 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:58.462881 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:56:58.463065 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:56:58.463447 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:56:58.463452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:56:58.463479 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:56:58.463949 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:56:58.463961 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.465247 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.465252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:56:58.467752 (+ 2500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:56:58.467767 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.469021 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.469030 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:56:58.469148 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:58.469221 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 02:56:58.469333 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 02:56:58.469456 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":959,"mutex_wait_us":65,"thread_start_us":721,"threads_started":1}
I20260620 02:56:58.470305 18060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:58.461792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46873 (local address 127.0.97.2:41645)
0620 02:56:58.463050 (+ 1258us) negotiation.cc:107] Waiting for socket to connect
0620 02:56:58.463074 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:56:58.463217 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:56:58.463706 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:56:58.463710 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:56:58.463740 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:56:58.464258 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:56:58.464269 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.465656 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.465662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:56:58.467757 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:56:58.467767 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.468928 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.468935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:56:58.469137 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:58.469181 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 02:56:58.469254 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 02:56:58.469399 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1107,"mutex_wait_us":9,"thread_start_us":561,"threads_started":1}
I20260620 02:56:58.470968 17718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9248cd6e1cb942fba8a12af94ac6fc44" candidate_uuid: "ee2dd6a4d5df4d93aa49baf1c98faa39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18902428d4b9433c974d2509bbfe736f" is_pre_election: true
I20260620 02:56:58.471352 17986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9248cd6e1cb942fba8a12af94ac6fc44" candidate_uuid: "ee2dd6a4d5df4d93aa49baf1c98faa39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc1d815d1e3448eea62052e286a1aca2" is_pre_election: true
I20260620 02:56:58.471613 17718 raft_consensus.cc:2454] T 9248cd6e1cb942fba8a12af94ac6fc44 P 18902428d4b9433c974d2509bbfe736f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee2dd6a4d5df4d93aa49baf1c98faa39 in term 0.
I20260620 02:56:58.471909 17986 raft_consensus.cc:2454] T 9248cd6e1cb942fba8a12af94ac6fc44 P dc1d815d1e3448eea62052e286a1aca2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee2dd6a4d5df4d93aa49baf1c98faa39 in term 0.
I20260620 02:56:58.472429 17787 leader_election.cc:302] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18902428d4b9433c974d2509bbfe736f, ee2dd6a4d5df4d93aa49baf1c98faa39; no voters:
I20260620 02:56:58.472833 18058 raft_consensus.cc:2781] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:56:58.472986 18058 raft_consensus.cc:482] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:56:58.473094 18058 raft_consensus.cc:3037] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:56:58.475308 18058 raft_consensus.cc:504] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1d815d1e3448eea62052e286a1aca2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46873 } } peers { permanent_uuid: "18902428d4b9433c974d2509bbfe736f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36647 } } peers { permanent_uuid: "ee2dd6a4d5df4d93aa49baf1c98faa39" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44333 } }
I20260620 02:56:58.475898 18058 leader_election.cc:288] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39 [CANDIDATE]: Term 1 election: Requested vote from peers dc1d815d1e3448eea62052e286a1aca2 (127.0.97.3:46873), 18902428d4b9433c974d2509bbfe736f (127.0.97.1:36647)
I20260620 02:56:58.476404 17718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9248cd6e1cb942fba8a12af94ac6fc44" candidate_uuid: "ee2dd6a4d5df4d93aa49baf1c98faa39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18902428d4b9433c974d2509bbfe736f"
I20260620 02:56:58.476403 17986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9248cd6e1cb942fba8a12af94ac6fc44" candidate_uuid: "ee2dd6a4d5df4d93aa49baf1c98faa39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc1d815d1e3448eea62052e286a1aca2"
I20260620 02:56:58.476622 17986 raft_consensus.cc:3037] T 9248cd6e1cb942fba8a12af94ac6fc44 P dc1d815d1e3448eea62052e286a1aca2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:56:58.476611 17718 raft_consensus.cc:3037] T 9248cd6e1cb942fba8a12af94ac6fc44 P 18902428d4b9433c974d2509bbfe736f [term 0 FOLLOWER]: Advancing to term 1
W20260620 02:56:58.477344 17764 tablet.cc:2369] T 9248cd6e1cb942fba8a12af94ac6fc44 P 18902428d4b9433c974d2509bbfe736f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:56:58.478986 17986 raft_consensus.cc:2454] T 9248cd6e1cb942fba8a12af94ac6fc44 P dc1d815d1e3448eea62052e286a1aca2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee2dd6a4d5df4d93aa49baf1c98faa39 in term 1.
I20260620 02:56:58.479380 17718 raft_consensus.cc:2454] T 9248cd6e1cb942fba8a12af94ac6fc44 P 18902428d4b9433c974d2509bbfe736f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee2dd6a4d5df4d93aa49baf1c98faa39 in term 1.
I20260620 02:56:58.479475 17785 leader_election.cc:302] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc1d815d1e3448eea62052e286a1aca2, ee2dd6a4d5df4d93aa49baf1c98faa39; no voters:
I20260620 02:56:58.480036 18058 raft_consensus.cc:2781] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:56:58.480530 18058 raft_consensus.cc:686] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39 [term 1 LEADER]: Becoming Leader. State: Replica: ee2dd6a4d5df4d93aa49baf1c98faa39, State: Running, Role: LEADER
I20260620 02:56:58.481020 18058 consensus_queue.cc:237] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1d815d1e3448eea62052e286a1aca2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46873 } } peers { permanent_uuid: "18902428d4b9433c974d2509bbfe736f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36647 } } peers { permanent_uuid: "ee2dd6a4d5df4d93aa49baf1c98faa39" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44333 } }
I20260620 02:56:58.485697 17574 catalog_manager.cc:5495] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee2dd6a4d5df4d93aa49baf1c98faa39 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "ee2dd6a4d5df4d93aa49baf1c98faa39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1d815d1e3448eea62052e286a1aca2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18902428d4b9433c974d2509bbfe736f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee2dd6a4d5df4d93aa49baf1c98faa39" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44333 } health_report { overall_health: HEALTHY } } }
W20260620 02:56:58.511405 17898 tablet.cc:2369] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:56:58.532485 18049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:58.526638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43918 (local address 127.0.97.2:44333)
0620 02:56:58.526895 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 02:56:58.526903 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:56:58.526947 (+ 44us) server_negotiation.cc:407] Connection header received
0620 02:56:58.527157 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:56:58.527160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:56:58.527238 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:56:58.527400 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:56:58.527811 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.528568 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.530018 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.530537 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.530636 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:58.531275 (+ 639us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:56:58.531782 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:56:58.531903 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 02:56:58.532087 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 02:56:58.532171 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
W20260620 02:56:58.536281 18032 tablet.cc:2369] T 9248cd6e1cb942fba8a12af94ac6fc44 P dc1d815d1e3448eea62052e286a1aca2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:56:58.541077 17986 raft_consensus.cc:1261] T 9248cd6e1cb942fba8a12af94ac6fc44 P dc1d815d1e3448eea62052e286a1aca2 [term 1 FOLLOWER]: Refusing update from remote peer ee2dd6a4d5df4d93aa49baf1c98faa39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:56:58.541079 17718 raft_consensus.cc:1261] T 9248cd6e1cb942fba8a12af94ac6fc44 P 18902428d4b9433c974d2509bbfe736f [term 1 FOLLOWER]: Refusing update from remote peer ee2dd6a4d5df4d93aa49baf1c98faa39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:56:58.542083 18058 consensus_queue.cc:1035] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18902428d4b9433c974d2509bbfe736f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:56:58.542424 18061 consensus_queue.cc:1035] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc1d815d1e3448eea62052e286a1aca2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:56:58.581182 17574 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:56:58.581406 17574 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:56:58.585224 17574 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e38ca8d711ee4066af3a2d4485a6e70e: Sending DeleteTablet for 3 replicas of tablet 9248cd6e1cb942fba8a12af94ac6fc44
I20260620 02:56:58.586707 17698 tablet_service.cc:1514] Processing DeleteTablet for tablet 9248cd6e1cb942fba8a12af94ac6fc44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:56:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48420
I20260620 02:56:58.586992 17966 tablet_service.cc:1514] Processing DeleteTablet for tablet 9248cd6e1cb942fba8a12af94ac6fc44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:56:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51452
I20260620 02:56:58.587939 17832 tablet_service.cc:1514] Processing DeleteTablet for tablet 9248cd6e1cb942fba8a12af94ac6fc44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:56:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43916
I20260620 02:56:58.588765 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 17635
I20260620 02:56:58.598961 18076 tablet_replica.cc:330] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39: stopping tablet replica
I20260620 02:56:58.599603 18076 raft_consensus.cc:2229] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:56:58.600380 18076 raft_consensus.cc:2258] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:56:58.601639 18074 tablet_replica.cc:330] T 9248cd6e1cb942fba8a12af94ac6fc44 P 18902428d4b9433c974d2509bbfe736f: stopping tablet replica
I20260620 02:56:58.602083 18074 raft_consensus.cc:2229] T 9248cd6e1cb942fba8a12af94ac6fc44 P 18902428d4b9433c974d2509bbfe736f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:56:58.602532 18075 tablet_replica.cc:330] T 9248cd6e1cb942fba8a12af94ac6fc44 P dc1d815d1e3448eea62052e286a1aca2: stopping tablet replica
I20260620 02:56:58.602628 18074 raft_consensus.cc:2258] T 9248cd6e1cb942fba8a12af94ac6fc44 P 18902428d4b9433c974d2509bbfe736f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:56:58.602937 18075 raft_consensus.cc:2229] T 9248cd6e1cb942fba8a12af94ac6fc44 P dc1d815d1e3448eea62052e286a1aca2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:56:58.603469 18075 raft_consensus.cc:2258] T 9248cd6e1cb942fba8a12af94ac6fc44 P dc1d815d1e3448eea62052e286a1aca2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:56:58.604468 18076 ts_tablet_manager.cc:1905] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:56:58.605178 18074 ts_tablet_manager.cc:1905] T 9248cd6e1cb942fba8a12af94ac6fc44 P 18902428d4b9433c974d2509bbfe736f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:56:58.605369 18075 ts_tablet_manager.cc:1905] T 9248cd6e1cb942fba8a12af94ac6fc44 P dc1d815d1e3448eea62052e286a1aca2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:56:58.611812 18076 ts_tablet_manager.cc:1918] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:56:58.611975 18076 log.cc:1196] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-1/wal/wals/9248cd6e1cb942fba8a12af94ac6fc44
I20260620 02:56:58.612214 18075 ts_tablet_manager.cc:1918] T 9248cd6e1cb942fba8a12af94ac6fc44 P dc1d815d1e3448eea62052e286a1aca2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Jun 20 02:56:58 dist-test-slave-8p8c krb5kdc[17529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:56:58.612366 18075 log.cc:1196] T 9248cd6e1cb942fba8a12af94ac6fc44 P dc1d815d1e3448eea62052e286a1aca2: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-2/wal/wals/9248cd6e1cb942fba8a12af94ac6fc44
I20260620 02:56:58.612537 18076 ts_tablet_manager.cc:1939] T 9248cd6e1cb942fba8a12af94ac6fc44 P ee2dd6a4d5df4d93aa49baf1c98faa39: Deleting consensus metadata
I20260620 02:56:58.612583 18074 ts_tablet_manager.cc:1918] T 9248cd6e1cb942fba8a12af94ac6fc44 P 18902428d4b9433c974d2509bbfe736f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:56:58.612815 18074 log.cc:1196] T 9248cd6e1cb942fba8a12af94ac6fc44 P 18902428d4b9433c974d2509bbfe736f: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781923914418765-388-0/minicluster-data/ts-0/wal/wals/9248cd6e1cb942fba8a12af94ac6fc44
I20260620 02:56:58.612926 18075 ts_tablet_manager.cc:1939] T 9248cd6e1cb942fba8a12af94ac6fc44 P dc1d815d1e3448eea62052e286a1aca2: Deleting consensus metadata
I20260620 02:56:58.613444 18074 ts_tablet_manager.cc:1939] T 9248cd6e1cb942fba8a12af94ac6fc44 P 18902428d4b9433c974d2509bbfe736f: Deleting consensus metadata
I20260620 02:56:58.614468 17559 catalog_manager.cc:4841] TS ee2dd6a4d5df4d93aa49baf1c98faa39 (127.0.97.2:44333): tablet 9248cd6e1cb942fba8a12af94ac6fc44 (table test-table [id=ef1f3f121cf14fc686e68c90b9060cf1]) successfully deleted
I20260620 02:56:58.615274 17560 catalog_manager.cc:4841] TS 18902428d4b9433c974d2509bbfe736f (127.0.97.1:36647): tablet 9248cd6e1cb942fba8a12af94ac6fc44 (table test-table [id=ef1f3f121cf14fc686e68c90b9060cf1]) successfully deleted
I20260620 02:56:58.615327 17558 catalog_manager.cc:4841] TS dc1d815d1e3448eea62052e286a1aca2 (127.0.97.3:46873): tablet 9248cd6e1cb942fba8a12af94ac6fc44 (table test-table [id=ef1f3f121cf14fc686e68c90b9060cf1]) successfully deleted
I20260620 02:56:58.621449 18046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:58.589821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48424 (local address 127.0.97.1:36647)
0620 02:56:58.590007 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 02:56:58.590014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:56:58.601255 (+ 11241us) server_negotiation.cc:407] Connection header received
0620 02:56:58.601602 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:56:58.601611 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:56:58.601769 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:56:58.601904 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:56:58.604205 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.605274 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.607275 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.607918 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.608010 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:58.615371 (+ 7361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:56:58.615385 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:56:58.615404 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:56:58.615449 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:56:58.618530 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:58.619435 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:58.619443 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:58.619446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:58.619584 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:58.620038 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:58.620041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:58.620043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:58.620438 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:56:58.620699 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 02:56:58.620994 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 02:56:58.621155 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 02:56:58.622140 17758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:56:58.713248 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 17635
I20260620 02:56:58.737802 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 17769
Jun 20 02:56:58 dist-test-slave-8p8c krb5kdc[17529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:56:58.756109 18049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:58.738741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43930 (local address 127.0.97.2:44333)
0620 02:56:58.739070 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0620 02:56:58.739074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:56:58.739097 (+ 23us) server_negotiation.cc:407] Connection header received
0620 02:56:58.739205 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:56:58.739208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:56:58.739273 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:56:58.739381 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:56:58.741077 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.742010 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.743976 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.744423 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.744531 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:58.749695 (+ 5164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:56:58.749709 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:56:58.749725 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:56:58.749768 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:56:58.753018 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:58.753759 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:58.753768 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:58.753770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:58.753920 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:58.754312 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:58.754327 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:58.754330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:58.754740 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:56:58.755110 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0620 02:56:58.755374 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 02:56:58.755600 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 02:56:58.757368 17892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:56:58.871857 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 17769
I20260620 02:56:58.887493 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 17903
Jun 20 02:56:58 dist-test-slave-8p8c krb5kdc[17529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:56:58.903928 18047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:58.888306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51456 (local address 127.0.97.3:46873)
0620 02:56:58.888483 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 02:56:58.888489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:56:58.888509 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:56:58.888572 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:56:58.888576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:56:58.888641 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:56:58.888745 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:56:58.890088 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.891001 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.892516 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:58.892915 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:58.893056 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:58.897859 (+ 4803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:56:58.897875 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:56:58.897892 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:56:58.897932 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:56:58.900748 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:58.901467 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:58.901475 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:58.901478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:58.901629 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:56:58.902020 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:56:58.902023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:56:58.902025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:56:58.902715 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:56:58.902933 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 02:56:58.903469 (+ 536us) server_negotiation.cc:299] Negotiation successful
0620 02:56:58.903675 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 02:56:58.904711 18026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:56:58.996677 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 17903
I20260620 02:56:59.015123 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 17542
I20260620 02:56:59.016506 17604 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:56:59.127135 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 17542
2026-06-20T02:56:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4418 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T02:56:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:56:59Z Disabled control of system clock
I20260620 02:56:59.204721 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44471
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 02:56:59.539012 18085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:56:59.539347 18085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:56:59.539494 18085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:56:59.547483 18085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:56:59.547609 18085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:56:59.547683 18085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:56:59.547773 18085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:56:59.547865 18085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:56:59.557289 18085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44471
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:56:59.558692 18085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:56:59.560524 18085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:56:59.572973 18090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:56:59.573066 18091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:56:59.573012 18093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:56:59.579159 18085 server_base.cc:1034] running on GCE node
I20260620 02:56:59.580019 18085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:56:59.581554 18085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:56:59.582755 18085 hybrid_clock.cc:648] HybridClock initialized: now 1781924219582712 us; error 31 us; skew 500 ppm
I20260620 02:56:59.583130 18085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:56:59.585456 18085 webserver.cc:460] Webserver started at http://127.0.97.62:39521/ using document root <none> and password file <none>
I20260620 02:56:59.586134 18085 fs_manager.cc:359] Metadata directory not provided
I20260620 02:56:59.586248 18085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:56:59.586640 18085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:56:59.589511 18085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "584ddec37d2743a3b7401613b33f40a0"
format_stamp: "Formatted at 2026-06-20 02:56:59 on dist-test-slave-8p8c"
server_key: "99c59f74fb405c515987f7cc7104af12"
server_key_iv: "71d248afaefeac5d4a48c265e6cf9943"
server_key_version: "encryptionkey@0"
I20260620 02:56:59.590296 18085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "584ddec37d2743a3b7401613b33f40a0"
format_stamp: "Formatted at 2026-06-20 02:56:59 on dist-test-slave-8p8c"
server_key: "99c59f74fb405c515987f7cc7104af12"
server_key_iv: "71d248afaefeac5d4a48c265e6cf9943"
server_key_version: "encryptionkey@0"
I20260620 02:56:59.595455 18085 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 02:56:59.598896 18099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:56:59.600273 18085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 02:56:59.600495 18085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "584ddec37d2743a3b7401613b33f40a0"
format_stamp: "Formatted at 2026-06-20 02:56:59 on dist-test-slave-8p8c"
server_key: "99c59f74fb405c515987f7cc7104af12"
server_key_iv: "71d248afaefeac5d4a48c265e6cf9943"
server_key_version: "encryptionkey@0"
I20260620 02:56:59.600714 18085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:56:59.623678 18085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:56:59.624519 18085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:56:59.624815 18085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:56:59.643682 18085 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44471
I20260620 02:56:59.643688 18150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44471 every 8 connection(s)
I20260620 02:56:59.645087 18085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:56:59.648908 18151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:56:59.648891 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 18085
I20260620 02:56:59.649111 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:56:59.649550 388 external_mini_cluster.cc:1428] Setting key b3efb55ed16a767b73addde65b2e8538
I20260620 02:56:59.664162 18151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 584ddec37d2743a3b7401613b33f40a0: Bootstrap starting.
I20260620 02:56:59.666811 18154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:56:59.651607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49128 (local address 127.0.97.62:44471)
0620 02:56:59.652742 (+ 1135us) server_negotiation.cc:206] Beginning negotiation
0620 02:56:59.652758 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 02:56:59.652810 (+ 52us) server_negotiation.cc:407] Connection header received
0620 02:56:59.655727 (+ 2917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:56:59.655760 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:56:59.656433 (+ 673us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:56:59.657063 (+ 630us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:56:59.657641 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:59.659573 (+ 1932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:59.661065 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:56:59.662197 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:56:59.662418 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:56:59.663718 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:56:59.663731 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:56:59.663752 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:56:59.664508 (+ 756us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:56:59.664630 (+ 122us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:56:59.664660 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:56:59.664790 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 02:56:59.665030 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 02:56:59.665384 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":361,"threads_started":1}
I20260620 02:56:59.668458 18151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 584ddec37d2743a3b7401613b33f40a0: Neither blocks nor log segments found. Creating new log.
I20260620 02:56:59.669602 18151 log.cc:824] T 00000000000000000000000000000000 P 584ddec37d2743a3b7401613b33f40a0: Log is configured to *not* fsync() on all Append() calls
I20260620 02:56:59.672880 18151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 584ddec37d2743a3b7401613b33f40a0: No bootstrap required, opened a new log
I20260620 02:56:59.680541 18151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 584ddec37d2743a3b7401613b33f40a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584ddec37d2743a3b7401613b33f40a0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44471 } }
I20260620 02:56:59.680925 18151 raft_consensus.cc:374] T 00000000000000000000000000000000 P 584ddec37d2743a3b7401613b33f40a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:56:59.681010 18151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 584ddec37d2743a3b7401613b33f40a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584ddec37d2743a3b7401613b33f40a0, State: Initialized, Role: FOLLOWER
I20260620 02:56:59.681542 18151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 584ddec37d2743a3b7401613b33f40a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584ddec37d2743a3b7401613b33f40a0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44471 } }
I20260620 02:56:59.681742 18151 raft_consensus.cc:388] T 00000000000000000000000000000000 P 584ddec37d2743a3b7401613b33f40a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:56:59.681838 18151 raft_consensus.cc:482] T 00000000000000000000000000000000 P 584ddec37d2743a3b7401613b33f40a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:56:59.681957 18151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 584ddec37d2743a3b7401613b33f40a0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:56:59.684146 18151 raft_consensus.cc:504] T 00000000000000000000000000000000 P 584ddec37d2743a3b7401613b33f40a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584ddec37d2743a3b7401613b33f40a0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44471 } }
I20260620 02:56:59.684603 18151 leader_election.cc:302] T 00000000000000000000000000000000 P 584ddec37d2743a3b7401613b33f40a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 584ddec37d2743a3b7401613b33f40a0; no voters:
I20260620 02:56:59.685235 18151 leader_election.cc:288] T 00000000000000000000000000000000 P 584ddec37d2743a3b7401613b33f40a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:56:59.685443 18156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 584ddec37d2743a3b7401613b33f40a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:56:59.687065 18156 raft_consensus.cc:686] T 00000000000000000000000000000000 P 584ddec37d2743a3b7401613b33f40a0 [term 1 LEADER]: Becoming Leader. State: Replica: 584ddec37d2743a3b7401613b33f40a0, State: Running, Role: LEADER
I20260620 02:56:59.687608 18156 consensus_queue.cc:237] T 00000000000000000000000000000000 P 584ddec37d2743a3b7401613b33f40a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584ddec37d2743a3b7401613b33f40a0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44471 } }
I20260620 02:56:59.687953 18151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 584ddec37d2743a3b7401613b33f40a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:56:59.690904 18157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 584ddec37d2743a3b7401613b33f40a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "584ddec37d2743a3b7401613b33f40a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584ddec37d2743a3b7401613b33f40a0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44471 } } }
I20260620 02:56:59.691438 18157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 584ddec37d2743a3b7401613b33f40a0 [sys.catalog]: This master's current role is: LEADER
I20260620 02:56:59.690891 18158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 584ddec37d2743a3b7401613b33f40a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 584ddec37d2743a3b7401613b33f40a0. Latest consensus state: current_term: 1 leader_uuid: "584ddec37d2743a3b7401613b33f40a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584ddec37d2743a3b7401613b33f40a0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44471 } } }
I20260620 02:56:59.691603 18158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 584ddec37d2743a3b7401613b33f40a0 [sys.catalog]: This master's current role is: LEADER
I20260620 02:56:59.693212 18162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:56:59.698855 18162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:56:59.708531 18162 catalog_manager.cc:1347] Generated new cluster ID: b30315e7f1104640b98e09c8e8ffeb33
I20260620 02:56:59.708676 18162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:56:59.732825 18162 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:56:59.734212 18162 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:56:59.746572 18162 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 584ddec37d2743a3b7401613b33f40a0: Generated new TSK 0
I20260620 02:56:59.747515 18162 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:56:59.760645 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44471
--builtin_ntp_servers=127.0.97.20:42581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 02:57:00.101739 18175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:00.102008 18175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:00.102157 18175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:00.109649 18175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:00.109745 18175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:00.109905 18175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:57:00.119221 18175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:00.120543 18175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:00.122417 18175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:00.136437 18180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:00.136399 18181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:00.136483 18183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:00.137151 18175 server_base.cc:1034] running on GCE node
I20260620 02:57:00.138049 18175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:00.139027 18175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:00.140252 18175 hybrid_clock.cc:648] HybridClock initialized: now 1781924220140201 us; error 37 us; skew 500 ppm
I20260620 02:57:00.140636 18175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:57:00.143249 18175 webserver.cc:460] Webserver started at http://127.0.97.1:43777/ using document root <none> and password file <none>
I20260620 02:57:00.143963 18175 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:00.144078 18175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:00.144426 18175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:00.147435 18175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-0/data/instance:
uuid: "e9a58779dcfa4b91b0e442d5c951c6e3"
format_stamp: "Formatted at 2026-06-20 02:57:00 on dist-test-slave-8p8c"
server_key: "48f3e93a95ceaacb1894c6e4b04f3579"
server_key_iv: "edf5b615f72d0efa4825b53d3c6bf4c0"
server_key_version: "encryptionkey@0"
I20260620 02:57:00.148267 18175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e9a58779dcfa4b91b0e442d5c951c6e3"
format_stamp: "Formatted at 2026-06-20 02:57:00 on dist-test-slave-8p8c"
server_key: "48f3e93a95ceaacb1894c6e4b04f3579"
server_key_iv: "edf5b615f72d0efa4825b53d3c6bf4c0"
server_key_version: "encryptionkey@0"
I20260620 02:57:00.153470 18175 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 02:57:00.157235 18189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:00.159065 18175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 02:57:00.159240 18175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-0/wal
uuid: "e9a58779dcfa4b91b0e442d5c951c6e3"
format_stamp: "Formatted at 2026-06-20 02:57:00 on dist-test-slave-8p8c"
server_key: "48f3e93a95ceaacb1894c6e4b04f3579"
server_key_iv: "edf5b615f72d0efa4825b53d3c6bf4c0"
server_key_version: "encryptionkey@0"
I20260620 02:57:00.159462 18175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:00.185699 18175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:00.186792 18175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:00.187110 18175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:00.188191 18175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:57:00.189936 18175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:57:00.190024 18175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:00.190135 18175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:57:00.190219 18175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:00.225545 18175 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39037
I20260620 02:57:00.225669 18301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39037 every 8 connection(s)
I20260620 02:57:00.226940 18175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:57:00.229882 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 18175
I20260620 02:57:00.230055 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:57:00.234458 388 external_mini_cluster.cc:1428] Setting key 62d9c310bfe480e132beecce9a651f53
I20260620 02:57:00.237270 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44471
--builtin_ntp_servers=127.0.97.20:42581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 02:57:00.241039 18304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:00.229628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44753 (local address 127.0.97.62:44471)
0620 02:57:00.230118 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:00.230123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:00.231626 (+ 1503us) server_negotiation.cc:407] Connection header received
0620 02:57:00.233624 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:00.233630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:00.233733 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:00.233867 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:00.235188 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:00.236298 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:00.238046 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:00.238549 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:00.238702 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:00.240183 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:00.240188 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:00.240191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:00.240208 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:00.240248 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:00.240260 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:00.240358 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:00.240599 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 02:57:00.240698 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":260,"threads_started":1}
I20260620 02:57:00.242224 18305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:00.230303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.1:44753)
0620 02:57:00.231434 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:00.231480 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:00.233309 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:00.234124 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:00.234132 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:00.234233 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:57:00.234998 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:00.235017 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:00.236473 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:00.236477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:00.237881 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:00.237894 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:00.238800 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:00.238810 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:00.239076 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:00.239834 (+ 758us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:57:00.239837 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:57:00.239879 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:57:00.239970 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:57:00.240356 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:57:00.240369 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:57:00.240379 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 02:57:00.240590 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 02:57:00.240998 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":364,"threads_started":1}
I20260620 02:57:00.244237 18302 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44471
I20260620 02:57:00.244675 18302 heartbeater.cc:464] Registering TS with master...
I20260620 02:57:00.245860 18302 heartbeater.cc:511] Master 127.0.97.62:44471 requested a full tablet report, sending...
I20260620 02:57:00.248261 18116 ts_manager.cc:194] Registered new tserver with Master: e9a58779dcfa4b91b0e442d5c951c6e3 (127.0.97.1:39037)
I20260620 02:57:00.249735 18116 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:44753
W20260620 02:57:00.579165 18306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:00.579463 18306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:00.579609 18306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:00.587486 18306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:00.587599 18306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:00.587764 18306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:57:00.597612 18306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:00.598969 18306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:00.600737 18306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:00.614085 18311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:00.614171 18312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:00.614145 18314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:00.615315 18306 server_base.cc:1034] running on GCE node
I20260620 02:57:00.615955 18306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:00.616876 18306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:00.618149 18306 hybrid_clock.cc:648] HybridClock initialized: now 1781924220618107 us; error 30 us; skew 500 ppm
I20260620 02:57:00.618589 18306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:57:00.621244 18306 webserver.cc:460] Webserver started at http://127.0.97.2:33437/ using document root <none> and password file <none>
I20260620 02:57:00.622009 18306 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:00.622131 18306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:00.622570 18306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:00.625622 18306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-1/data/instance:
uuid: "064c105c77bf473e8daa8db462e815fb"
format_stamp: "Formatted at 2026-06-20 02:57:00 on dist-test-slave-8p8c"
server_key: "1f5bc44e953cea8ac5e5383c75f0b517"
server_key_iv: "125e3b49d37db12b179ddd2b57114e5d"
server_key_version: "encryptionkey@0"
I20260620 02:57:00.626483 18306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-1/wal/instance:
uuid: "064c105c77bf473e8daa8db462e815fb"
format_stamp: "Formatted at 2026-06-20 02:57:00 on dist-test-slave-8p8c"
server_key: "1f5bc44e953cea8ac5e5383c75f0b517"
server_key_iv: "125e3b49d37db12b179ddd2b57114e5d"
server_key_version: "encryptionkey@0"
I20260620 02:57:00.631914 18306 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 02:57:00.635649 18320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:00.637199 18306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 02:57:00.637384 18306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-1/wal
uuid: "064c105c77bf473e8daa8db462e815fb"
format_stamp: "Formatted at 2026-06-20 02:57:00 on dist-test-slave-8p8c"
server_key: "1f5bc44e953cea8ac5e5383c75f0b517"
server_key_iv: "125e3b49d37db12b179ddd2b57114e5d"
server_key_version: "encryptionkey@0"
I20260620 02:57:00.637602 18306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:00.656170 18306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:00.657037 18306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:00.657338 18306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:00.658299 18306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:57:00.659821 18306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:57:00.659940 18306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:00.660076 18306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:57:00.660177 18306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:00.697520 18306 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40253
I20260620 02:57:00.697613 18432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40253 every 8 connection(s)
I20260620 02:57:00.698917 18306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:57:00.704186 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 18306
I20260620 02:57:00.704391 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:57:00.704991 388 external_mini_cluster.cc:1428] Setting key 3571ee64bf16c0a0efcf12165fda9f3d
I20260620 02:57:00.708539 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44471
--builtin_ntp_servers=127.0.97.20:42581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 02:57:00.715483 18304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:00.701746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49139 (local address 127.0.97.62:44471)
0620 02:57:00.702023 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:00.702028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:00.703845 (+ 1817us) server_negotiation.cc:407] Connection header received
0620 02:57:00.706468 (+ 2623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:00.706473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:00.706565 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:00.706665 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:00.708143 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:00.709077 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:00.711967 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:00.712541 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:00.712674 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:00.714605 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:00.714610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:00.714614 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:00.714635 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:00.714669 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:00.714681 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:00.714784 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:00.715025 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 02:57:00.715157 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260620 02:57:00.716588 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:00.702318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.2:49139)
0620 02:57:00.703644 (+ 1326us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:00.703685 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:00.705995 (+ 2310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:00.706958 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:00.706971 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:00.707109 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:57:00.707939 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:00.707968 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:00.709262 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:00.709271 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:00.711786 (+ 2515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:00.711805 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:00.712799 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:00.712808 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:00.713175 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:00.714132 (+ 957us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:57:00.714139 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:57:00.714208 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:57:00.714330 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:57:00.714782 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:57:00.714794 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:57:00.714805 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 02:57:00.715000 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 02:57:00.715464 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"thread_start_us":369,"threads_started":1}
I20260620 02:57:00.718407 18433 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44471
I20260620 02:57:00.718761 18433 heartbeater.cc:464] Registering TS with master...
I20260620 02:57:00.719720 18433 heartbeater.cc:511] Master 127.0.97.62:44471 requested a full tablet report, sending...
I20260620 02:57:00.721422 18116 ts_manager.cc:194] Registered new tserver with Master: 064c105c77bf473e8daa8db462e815fb (127.0.97.2:40253)
I20260620 02:57:00.722256 18116 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:49139
W20260620 02:57:01.048617 18436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:01.048893 18436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:01.049045 18436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:01.056229 18436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:01.056326 18436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:01.056504 18436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:57:01.065544 18436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:01.066844 18436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:01.068621 18436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:01.081861 18444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:01.081835 18442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:01.082305 18441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:01.082736 18436 server_base.cc:1034] running on GCE node
I20260620 02:57:01.083284 18436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:01.084192 18436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:01.085418 18436 hybrid_clock.cc:648] HybridClock initialized: now 1781924221085366 us; error 38 us; skew 500 ppm
I20260620 02:57:01.085708 18436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:57:01.088009 18436 webserver.cc:460] Webserver started at http://127.0.97.3:42087/ using document root <none> and password file <none>
I20260620 02:57:01.088683 18436 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:01.088768 18436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:01.089033 18436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:01.091845 18436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-2/data/instance:
uuid: "0be3db2c50b7476da6339eff9de94b79"
format_stamp: "Formatted at 2026-06-20 02:57:01 on dist-test-slave-8p8c"
server_key: "947e6899f32d27e73a9f821062c4d008"
server_key_iv: "158736268f6b9d7a09a53d06d08d1b5d"
server_key_version: "encryptionkey@0"
I20260620 02:57:01.092552 18436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-2/wal/instance:
uuid: "0be3db2c50b7476da6339eff9de94b79"
format_stamp: "Formatted at 2026-06-20 02:57:01 on dist-test-slave-8p8c"
server_key: "947e6899f32d27e73a9f821062c4d008"
server_key_iv: "158736268f6b9d7a09a53d06d08d1b5d"
server_key_version: "encryptionkey@0"
I20260620 02:57:01.097609 18436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 02:57:01.101037 18450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:01.102572 18436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:57:01.102758 18436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-2/wal
uuid: "0be3db2c50b7476da6339eff9de94b79"
format_stamp: "Formatted at 2026-06-20 02:57:01 on dist-test-slave-8p8c"
server_key: "947e6899f32d27e73a9f821062c4d008"
server_key_iv: "158736268f6b9d7a09a53d06d08d1b5d"
server_key_version: "encryptionkey@0"
I20260620 02:57:01.103013 18436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:01.124720 18436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:01.125617 18436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:01.125926 18436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:01.127085 18436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:57:01.128722 18436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:57:01.128854 18436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:01.129015 18436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:57:01.129112 18436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:01.165692 18436 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46835
I20260620 02:57:01.165782 18562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46835 every 8 connection(s)
I20260620 02:57:01.167145 18436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:57:01.177337 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 18436
I20260620 02:57:01.177587 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:57:01.178067 388 external_mini_cluster.cc:1428] Setting key be5442b3d9070dcd10b5a83a48eefa22
I20260620 02:57:01.180789 18304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:01.169971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40807 (local address 127.0.97.62:44471)
0620 02:57:01.170200 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:01.170206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:01.171736 (+ 1530us) server_negotiation.cc:407] Connection header received
0620 02:57:01.173493 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:01.173498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:01.173566 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:01.173665 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:01.174886 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.175626 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.177513 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.178063 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.178229 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:01.179813 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:01.179820 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:01.179824 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:01.179851 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:01.179892 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:01.179905 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:01.179981 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:01.180288 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 02:57:01.180400 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 02:57:01.181794 18565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:01.170554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44471 (local address 127.0.97.3:40807)
0620 02:57:01.171567 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:01.171612 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:01.173219 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:01.173921 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:01.173937 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:01.174036 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:57:01.174723 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:01.174741 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.175775 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.175779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:01.177271 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:01.177287 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.178328 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.178335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:01.178694 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:01.179491 (+ 797us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:57:01.179493 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:57:01.179537 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:57:01.179633 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:57:01.180024 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:57:01.180036 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:57:01.180046 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 02:57:01.180234 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 02:57:01.180571 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":351,"threads_started":1}
I20260620 02:57:01.183177 18563 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44471
I20260620 02:57:01.183511 18563 heartbeater.cc:464] Registering TS with master...
I20260620 02:57:01.184363 18563 heartbeater.cc:511] Master 127.0.97.62:44471 requested a full tablet report, sending...
I20260620 02:57:01.185818 18116 ts_manager.cc:194] Registered new tserver with Master: 0be3db2c50b7476da6339eff9de94b79 (127.0.97.3:46835)
I20260620 02:57:01.186748 18116 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:40807
I20260620 02:57:01.193670 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:57:01.204053 18304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:01.198253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49130 (local address 127.0.97.62:44471)
0620 02:57:01.198524 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:01.198531 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:01.198831 (+ 300us) server_negotiation.cc:407] Connection header received
0620 02:57:01.198958 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:01.198962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:01.199039 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:01.199136 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:01.199539 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.200304 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.201763 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.202200 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.202298 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:01.203376 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:01.203381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:01.203384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:01.203401 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:01.203433 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:01.203446 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:01.203521 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:01.203643 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 02:57:01.203719 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 02:57:01.208185 18116 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:57:01.215857 18116 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:57:01.252210 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:01.228642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40253 (local address 127.0.0.1:48800)
0620 02:57:01.240678 (+ 12036us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:01.240732 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:01.240954 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:01.242747 (+ 1793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:01.242752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:01.242779 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:57:01.243221 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:01.243234 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.245455 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.245461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:01.248495 (+ 3034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:01.248510 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.251346 (+ 2836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.251355 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:01.251523 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:01.251556 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 02:57:01.251689 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 02:57:01.251883 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11690,"thread_start_us":20739,"threads_started":1}
I20260620 02:57:01.252246 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:01.229082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48800 (local address 127.0.97.2:40253)
0620 02:57:01.238781 (+ 9699us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:01.238797 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:01.241228 (+ 2431us) server_negotiation.cc:407] Connection header received
0620 02:57:01.241352 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:01.241364 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:01.241667 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:01.241868 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:57:01.243391 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.245294 (+ 1903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.248728 (+ 3434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.250566 (+ 1838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.251072 (+ 506us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:01.251341 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:01.251743 (+ 402us) server_negotiation.cc:299] Negotiation successful
0620 02:57:01.251912 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9449,"thread_start_us":9292,"threads_started":1}
I20260620 02:57:01.252779 18302 heartbeater.cc:503] Master 127.0.97.62:44471 was elected leader, sending a full tablet report...
I20260620 02:57:01.255461 18578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:01.228988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39037 (local address 127.0.0.1:41294)
0620 02:57:01.241679 (+ 12691us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:01.241712 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:01.241948 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:01.246643 (+ 4695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:01.246648 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:01.246695 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:57:01.247170 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:01.247184 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.249235 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.249244 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:01.251804 (+ 2560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:01.251820 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.254566 (+ 2746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.254573 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:01.254842 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:01.254883 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 02:57:01.254976 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 02:57:01.255105 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11664,"mutex_wait_us":30,"thread_start_us":13098,"threads_started":1}
I20260620 02:57:01.255623 18579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:01.229623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41294 (local address 127.0.97.1:39037)
0620 02:57:01.245962 (+ 16339us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:01.245974 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:01.246005 (+ 31us) server_negotiation.cc:407] Connection header received
0620 02:57:01.246113 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:01.246122 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:01.246414 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:01.247407 (+ 993us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:57:01.247629 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.249105 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.252041 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.254353 (+ 2312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.254531 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:01.254832 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:01.255034 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 02:57:01.255273 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16168,"thread_start_us":16069,"threads_started":1}
I20260620 02:57:01.256817 18368 tablet_service.cc:1467] Processing CreateTablet for tablet 9263d97cbb774630a7c62957e1cdd77d (DEFAULT_TABLE table=test-table [id=4424d59c38244cc28c26bd452a8273ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:57:01.258282 18368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9263d97cbb774630a7c62957e1cdd77d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:01.258782 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:01.229337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46835 (local address 127.0.0.1:49900)
0620 02:57:01.249718 (+ 20381us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:01.249747 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:01.249888 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:01.250712 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:01.250718 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:01.250744 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:57:01.251153 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:01.251165 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.253711 (+ 2546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.253715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:01.256288 (+ 2573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:01.256304 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.257902 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.257909 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:01.258101 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:01.258136 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 02:57:01.258242 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 02:57:01.258426 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20241,"mutex_wait_us":24,"thread_start_us":11419,"threads_started":1}
I20260620 02:57:01.258797 18580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:01.229653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49900 (local address 127.0.97.3:46835)
0620 02:57:01.242640 (+ 12987us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:01.242647 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:01.249875 (+ 7228us) server_negotiation.cc:407] Connection header received
0620 02:57:01.250040 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:01.250048 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:01.250279 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:01.250491 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:57:01.251382 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.253239 (+ 1857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.256482 (+ 3243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.257693 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.257840 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:01.258109 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:01.258285 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 02:57:01.258464 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12809,"thread_start_us":12634,"threads_started":1}
I20260620 02:57:01.259486 18237 tablet_service.cc:1467] Processing CreateTablet for tablet 9263d97cbb774630a7c62957e1cdd77d (DEFAULT_TABLE table=test-table [id=4424d59c38244cc28c26bd452a8273ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:57:01.260738 18237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9263d97cbb774630a7c62957e1cdd77d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:01.262408 18498 tablet_service.cc:1467] Processing CreateTablet for tablet 9263d97cbb774630a7c62957e1cdd77d (DEFAULT_TABLE table=test-table [id=4424d59c38244cc28c26bd452a8273ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:57:01.263808 18498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9263d97cbb774630a7c62957e1cdd77d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:01.271996 18581 tablet_bootstrap.cc:492] T 9263d97cbb774630a7c62957e1cdd77d P 064c105c77bf473e8daa8db462e815fb: Bootstrap starting.
I20260620 02:57:01.275281 18581 tablet_bootstrap.cc:654] T 9263d97cbb774630a7c62957e1cdd77d P 064c105c77bf473e8daa8db462e815fb: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:01.276441 18582 tablet_bootstrap.cc:492] T 9263d97cbb774630a7c62957e1cdd77d P e9a58779dcfa4b91b0e442d5c951c6e3: Bootstrap starting.
I20260620 02:57:01.276724 18581 log.cc:824] T 9263d97cbb774630a7c62957e1cdd77d P 064c105c77bf473e8daa8db462e815fb: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:01.287648 18582 tablet_bootstrap.cc:654] T 9263d97cbb774630a7c62957e1cdd77d P e9a58779dcfa4b91b0e442d5c951c6e3: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:01.289637 18582 log.cc:824] T 9263d97cbb774630a7c62957e1cdd77d P e9a58779dcfa4b91b0e442d5c951c6e3: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:01.292086 18584 tablet_bootstrap.cc:492] T 9263d97cbb774630a7c62957e1cdd77d P 0be3db2c50b7476da6339eff9de94b79: Bootstrap starting.
I20260620 02:57:01.295357 18582 tablet_bootstrap.cc:492] T 9263d97cbb774630a7c62957e1cdd77d P e9a58779dcfa4b91b0e442d5c951c6e3: No bootstrap required, opened a new log
I20260620 02:57:01.295615 18581 tablet_bootstrap.cc:492] T 9263d97cbb774630a7c62957e1cdd77d P 064c105c77bf473e8daa8db462e815fb: No bootstrap required, opened a new log
I20260620 02:57:01.295692 18582 ts_tablet_manager.cc:1397] T 9263d97cbb774630a7c62957e1cdd77d P e9a58779dcfa4b91b0e442d5c951c6e3: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.002s
I20260620 02:57:01.295914 18581 ts_tablet_manager.cc:1397] T 9263d97cbb774630a7c62957e1cdd77d P 064c105c77bf473e8daa8db462e815fb: Time spent bootstrapping tablet: real 0.024s user 0.013s sys 0.008s
I20260620 02:57:01.297194 18584 tablet_bootstrap.cc:654] T 9263d97cbb774630a7c62957e1cdd77d P 0be3db2c50b7476da6339eff9de94b79: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:01.298589 18584 log.cc:824] T 9263d97cbb774630a7c62957e1cdd77d P 0be3db2c50b7476da6339eff9de94b79: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:01.301879 18584 tablet_bootstrap.cc:492] T 9263d97cbb774630a7c62957e1cdd77d P 0be3db2c50b7476da6339eff9de94b79: No bootstrap required, opened a new log
I20260620 02:57:01.302237 18584 ts_tablet_manager.cc:1397] T 9263d97cbb774630a7c62957e1cdd77d P 0be3db2c50b7476da6339eff9de94b79: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260620 02:57:01.304253 18582 raft_consensus.cc:348] T 9263d97cbb774630a7c62957e1cdd77d P e9a58779dcfa4b91b0e442d5c951c6e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064c105c77bf473e8daa8db462e815fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40253 } } peers { permanent_uuid: "e9a58779dcfa4b91b0e442d5c951c6e3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39037 } } peers { permanent_uuid: "0be3db2c50b7476da6339eff9de94b79" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46835 } }
I20260620 02:57:01.304633 18582 raft_consensus.cc:374] T 9263d97cbb774630a7c62957e1cdd77d P e9a58779dcfa4b91b0e442d5c951c6e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:01.304773 18582 raft_consensus.cc:729] T 9263d97cbb774630a7c62957e1cdd77d P e9a58779dcfa4b91b0e442d5c951c6e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9a58779dcfa4b91b0e442d5c951c6e3, State: Initialized, Role: FOLLOWER
I20260620 02:57:01.305425 18582 consensus_queue.cc:260] T 9263d97cbb774630a7c62957e1cdd77d P e9a58779dcfa4b91b0e442d5c951c6e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064c105c77bf473e8daa8db462e815fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40253 } } peers { permanent_uuid: "e9a58779dcfa4b91b0e442d5c951c6e3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39037 } } peers { permanent_uuid: "0be3db2c50b7476da6339eff9de94b79" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46835 } }
I20260620 02:57:01.307255 18582 ts_tablet_manager.cc:1428] T 9263d97cbb774630a7c62957e1cdd77d P e9a58779dcfa4b91b0e442d5c951c6e3: Time spent starting tablet: real 0.011s user 0.006s sys 0.006s
I20260620 02:57:01.307336 18581 raft_consensus.cc:348] T 9263d97cbb774630a7c62957e1cdd77d P 064c105c77bf473e8daa8db462e815fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064c105c77bf473e8daa8db462e815fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40253 } } peers { permanent_uuid: "e9a58779dcfa4b91b0e442d5c951c6e3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39037 } } peers { permanent_uuid: "0be3db2c50b7476da6339eff9de94b79" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46835 } }
I20260620 02:57:01.307984 18581 raft_consensus.cc:374] T 9263d97cbb774630a7c62957e1cdd77d P 064c105c77bf473e8daa8db462e815fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:01.308099 18581 raft_consensus.cc:729] T 9263d97cbb774630a7c62957e1cdd77d P 064c105c77bf473e8daa8db462e815fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 064c105c77bf473e8daa8db462e815fb, State: Initialized, Role: FOLLOWER
I20260620 02:57:01.308782 18581 consensus_queue.cc:260] T 9263d97cbb774630a7c62957e1cdd77d P 064c105c77bf473e8daa8db462e815fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064c105c77bf473e8daa8db462e815fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40253 } } peers { permanent_uuid: "e9a58779dcfa4b91b0e442d5c951c6e3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39037 } } peers { permanent_uuid: "0be3db2c50b7476da6339eff9de94b79" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46835 } }
I20260620 02:57:01.310000 18433 heartbeater.cc:503] Master 127.0.97.62:44471 was elected leader, sending a full tablet report...
I20260620 02:57:01.310702 18581 ts_tablet_manager.cc:1428] T 9263d97cbb774630a7c62957e1cdd77d P 064c105c77bf473e8daa8db462e815fb: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260620 02:57:01.313984 18584 raft_consensus.cc:348] T 9263d97cbb774630a7c62957e1cdd77d P 0be3db2c50b7476da6339eff9de94b79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064c105c77bf473e8daa8db462e815fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40253 } } peers { permanent_uuid: "e9a58779dcfa4b91b0e442d5c951c6e3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39037 } } peers { permanent_uuid: "0be3db2c50b7476da6339eff9de94b79" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46835 } }
I20260620 02:57:01.314533 18584 raft_consensus.cc:374] T 9263d97cbb774630a7c62957e1cdd77d P 0be3db2c50b7476da6339eff9de94b79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:01.314662 18584 raft_consensus.cc:729] T 9263d97cbb774630a7c62957e1cdd77d P 0be3db2c50b7476da6339eff9de94b79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0be3db2c50b7476da6339eff9de94b79, State: Initialized, Role: FOLLOWER
I20260620 02:57:01.315336 18584 consensus_queue.cc:260] T 9263d97cbb774630a7c62957e1cdd77d P 0be3db2c50b7476da6339eff9de94b79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064c105c77bf473e8daa8db462e815fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40253 } } peers { permanent_uuid: "e9a58779dcfa4b91b0e442d5c951c6e3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39037 } } peers { permanent_uuid: "0be3db2c50b7476da6339eff9de94b79" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46835 } }
I20260620 02:57:01.315613 18588 raft_consensus.cc:482] T 9263d97cbb774630a7c62957e1cdd77d P 064c105c77bf473e8daa8db462e815fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:57:01.315860 18588 raft_consensus.cc:504] T 9263d97cbb774630a7c62957e1cdd77d P 064c105c77bf473e8daa8db462e815fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064c105c77bf473e8daa8db462e815fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40253 } } peers { permanent_uuid: "e9a58779dcfa4b91b0e442d5c951c6e3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39037 } } peers { permanent_uuid: "0be3db2c50b7476da6339eff9de94b79" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46835 } }
I20260620 02:57:01.316609 18563 heartbeater.cc:503] Master 127.0.97.62:44471 was elected leader, sending a full tablet report...
I20260620 02:57:01.317652 18588 leader_election.cc:288] T 9263d97cbb774630a7c62957e1cdd77d P 064c105c77bf473e8daa8db462e815fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9a58779dcfa4b91b0e442d5c951c6e3 (127.0.97.1:39037), 0be3db2c50b7476da6339eff9de94b79 (127.0.97.3:46835)
I20260620 02:57:01.320673 18584 ts_tablet_manager.cc:1428] T 9263d97cbb774630a7c62957e1cdd77d P 0be3db2c50b7476da6339eff9de94b79: Time spent starting tablet: real 0.018s user 0.010s sys 0.004s
I20260620 02:57:01.325541 18579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:01.318628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47045 (local address 127.0.97.1:39037)
0620 02:57:01.319193 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:01.319200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:01.319221 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:57:01.319295 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:01.319303 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:01.319395 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:01.319501 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:57:01.320321 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.321216 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.323851 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.324671 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.324829 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:01.324932 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:01.325245 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 02:57:01.325349 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393}
I20260620 02:57:01.325649 18590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:01.318030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39037 (local address 127.0.97.2:47045)
0620 02:57:01.318694 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:01.318714 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:01.318848 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:01.319521 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:01.319527 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:01.319554 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:57:01.320110 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:01.320122 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.321338 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.321344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:01.323657 (+ 2313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:01.323672 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.324872 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.324879 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:01.325041 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:01.325069 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 02:57:01.325196 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 02:57:01.325327 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":653,"threads_started":1}
I20260620 02:57:01.326932 18257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9263d97cbb774630a7c62957e1cdd77d" candidate_uuid: "064c105c77bf473e8daa8db462e815fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9a58779dcfa4b91b0e442d5c951c6e3" is_pre_election: true
I20260620 02:57:01.327451 18580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:01.318092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35793 (local address 127.0.97.3:46835)
0620 02:57:01.318313 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:01.318322 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:01.320462 (+ 2140us) server_negotiation.cc:407] Connection header received
0620 02:57:01.320729 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:01.320735 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:01.320803 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:01.320895 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:57:01.322231 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.323353 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.325187 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.326354 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.326524 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:01.326593 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:01.327014 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 02:57:01.327115 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":63}
I20260620 02:57:01.327454 18591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:01.319561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46835 (local address 127.0.97.2:35793)
0620 02:57:01.320358 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:01.320380 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:01.320577 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:01.321357 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:01.321362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:01.321396 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:57:01.321970 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:01.321985 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.323494 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.323499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:01.325005 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:01.325017 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.326585 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.326592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:01.326803 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:01.326843 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 02:57:01.326956 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 02:57:01.327097 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":1,"thread_start_us":447,"threads_started":1}
I20260620 02:57:01.327580 18257 raft_consensus.cc:2454] T 9263d97cbb774630a7c62957e1cdd77d P e9a58779dcfa4b91b0e442d5c951c6e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 064c105c77bf473e8daa8db462e815fb in term 0.
I20260620 02:57:01.328321 18321 leader_election.cc:302] T 9263d97cbb774630a7c62957e1cdd77d P 064c105c77bf473e8daa8db462e815fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 064c105c77bf473e8daa8db462e815fb, e9a58779dcfa4b91b0e442d5c951c6e3; no voters:
I20260620 02:57:01.328764 18588 raft_consensus.cc:2781] T 9263d97cbb774630a7c62957e1cdd77d P 064c105c77bf473e8daa8db462e815fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:57:01.328593 18518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9263d97cbb774630a7c62957e1cdd77d" candidate_uuid: "064c105c77bf473e8daa8db462e815fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0be3db2c50b7476da6339eff9de94b79" is_pre_election: true
I20260620 02:57:01.328943 18588 raft_consensus.cc:482] T 9263d97cbb774630a7c62957e1cdd77d P 064c105c77bf473e8daa8db462e815fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:57:01.329057 18588 raft_consensus.cc:3037] T 9263d97cbb774630a7c62957e1cdd77d P 064c105c77bf473e8daa8db462e815fb [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:01.329311 18518 raft_consensus.cc:2454] T 9263d97cbb774630a7c62957e1cdd77d P 0be3db2c50b7476da6339eff9de94b79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 064c105c77bf473e8daa8db462e815fb in term 0.
I20260620 02:57:01.331691 18588 raft_consensus.cc:504] T 9263d97cbb774630a7c62957e1cdd77d P 064c105c77bf473e8daa8db462e815fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064c105c77bf473e8daa8db462e815fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40253 } } peers { permanent_uuid: "e9a58779dcfa4b91b0e442d5c951c6e3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39037 } } peers { permanent_uuid: "0be3db2c50b7476da6339eff9de94b79" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46835 } }
I20260620 02:57:01.332497 18588 leader_election.cc:288] T 9263d97cbb774630a7c62957e1cdd77d P 064c105c77bf473e8daa8db462e815fb [CANDIDATE]: Term 1 election: Requested vote from peers e9a58779dcfa4b91b0e442d5c951c6e3 (127.0.97.1:39037), 0be3db2c50b7476da6339eff9de94b79 (127.0.97.3:46835)
I20260620 02:57:01.333068 18257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9263d97cbb774630a7c62957e1cdd77d" candidate_uuid: "064c105c77bf473e8daa8db462e815fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9a58779dcfa4b91b0e442d5c951c6e3"
I20260620 02:57:01.333067 18518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9263d97cbb774630a7c62957e1cdd77d" candidate_uuid: "064c105c77bf473e8daa8db462e815fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0be3db2c50b7476da6339eff9de94b79"
I20260620 02:57:01.333285 18518 raft_consensus.cc:3037] T 9263d97cbb774630a7c62957e1cdd77d P 0be3db2c50b7476da6339eff9de94b79 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:01.333287 18257 raft_consensus.cc:3037] T 9263d97cbb774630a7c62957e1cdd77d P e9a58779dcfa4b91b0e442d5c951c6e3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:01.336588 18518 raft_consensus.cc:2454] T 9263d97cbb774630a7c62957e1cdd77d P 0be3db2c50b7476da6339eff9de94b79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 064c105c77bf473e8daa8db462e815fb in term 1.
I20260620 02:57:01.336588 18257 raft_consensus.cc:2454] T 9263d97cbb774630a7c62957e1cdd77d P e9a58779dcfa4b91b0e442d5c951c6e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 064c105c77bf473e8daa8db462e815fb in term 1.
I20260620 02:57:01.337057 18323 leader_election.cc:302] T 9263d97cbb774630a7c62957e1cdd77d P 064c105c77bf473e8daa8db462e815fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 064c105c77bf473e8daa8db462e815fb, 0be3db2c50b7476da6339eff9de94b79; no voters:
I20260620 02:57:01.337458 18588 raft_consensus.cc:2781] T 9263d97cbb774630a7c62957e1cdd77d P 064c105c77bf473e8daa8db462e815fb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:57:01.337980 18588 raft_consensus.cc:686] T 9263d97cbb774630a7c62957e1cdd77d P 064c105c77bf473e8daa8db462e815fb [term 1 LEADER]: Becoming Leader. State: Replica: 064c105c77bf473e8daa8db462e815fb, State: Running, Role: LEADER
I20260620 02:57:01.338621 18588 consensus_queue.cc:237] T 9263d97cbb774630a7c62957e1cdd77d P 064c105c77bf473e8daa8db462e815fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064c105c77bf473e8daa8db462e815fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40253 } } peers { permanent_uuid: "e9a58779dcfa4b91b0e442d5c951c6e3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39037 } } peers { permanent_uuid: "0be3db2c50b7476da6339eff9de94b79" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46835 } }
I20260620 02:57:01.343623 18116 catalog_manager.cc:5495] T 9263d97cbb774630a7c62957e1cdd77d P 064c105c77bf473e8daa8db462e815fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 064c105c77bf473e8daa8db462e815fb (127.0.97.2). New cstate: current_term: 1 leader_uuid: "064c105c77bf473e8daa8db462e815fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064c105c77bf473e8daa8db462e815fb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9a58779dcfa4b91b0e442d5c951c6e3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0be3db2c50b7476da6339eff9de94b79" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46835 } health_report { overall_health: UNKNOWN } } }
I20260620 02:57:01.375411 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:01.369816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48804 (local address 127.0.97.2:40253)
0620 02:57:01.370053 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:01.370059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:01.370080 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:57:01.370194 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:01.370198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:01.370292 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:01.370492 (+ 200us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:57:01.370898 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.371648 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.373008 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.373448 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.373553 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:01.374066 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:57:01.374609 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:57:01.374742 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:01.374914 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 02:57:01.375121 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 02:57:01.383967 18518 raft_consensus.cc:1261] T 9263d97cbb774630a7c62957e1cdd77d P 0be3db2c50b7476da6339eff9de94b79 [term 1 FOLLOWER]: Refusing update from remote peer 064c105c77bf473e8daa8db462e815fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:57:01.383966 18257 raft_consensus.cc:1261] T 9263d97cbb774630a7c62957e1cdd77d P e9a58779dcfa4b91b0e442d5c951c6e3 [term 1 FOLLOWER]: Refusing update from remote peer 064c105c77bf473e8daa8db462e815fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:57:01.384977 18592 consensus_queue.cc:1035] T 9263d97cbb774630a7c62957e1cdd77d P 064c105c77bf473e8daa8db462e815fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a58779dcfa4b91b0e442d5c951c6e3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:57:01.385329 18588 consensus_queue.cc:1035] T 9263d97cbb774630a7c62957e1cdd77d P 064c105c77bf473e8daa8db462e815fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0be3db2c50b7476da6339eff9de94b79" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:57:01.397960 18595 mvcc.cc:205] Tried to move back new op lower bound from 7298761610778374144 to 7298761610607431680. Current Snapshot: MvccSnapshot[applied={T|T < 7298761610778374144}]
I20260620 02:57:01.415501 18115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:57:01.415819 18115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:57:01.419608 18115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 584ddec37d2743a3b7401613b33f40a0: Sending DeleteTablet for 3 replicas of tablet 9263d97cbb774630a7c62957e1cdd77d
I20260620 02:57:01.420799 18237 tablet_service.cc:1514] Processing DeleteTablet for tablet 9263d97cbb774630a7c62957e1cdd77d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:57:01 UTC) from {username='slave'} at 127.0.0.1:41294
I20260620 02:57:01.421119 18498 tablet_service.cc:1514] Processing DeleteTablet for tablet 9263d97cbb774630a7c62957e1cdd77d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:57:01 UTC) from {username='slave'} at 127.0.0.1:49900
I20260620 02:57:01.421154 18368 tablet_service.cc:1514] Processing DeleteTablet for tablet 9263d97cbb774630a7c62957e1cdd77d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:57:01 UTC) from {username='slave'} at 127.0.0.1:48800
I20260620 02:57:01.422708 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 18175
I20260620 02:57:01.433404 18607 tablet_replica.cc:330] T 9263d97cbb774630a7c62957e1cdd77d P 064c105c77bf473e8daa8db462e815fb: stopping tablet replica
I20260620 02:57:01.434070 18606 tablet_replica.cc:330] T 9263d97cbb774630a7c62957e1cdd77d P 0be3db2c50b7476da6339eff9de94b79: stopping tablet replica
I20260620 02:57:01.434954 18607 raft_consensus.cc:2229] T 9263d97cbb774630a7c62957e1cdd77d P 064c105c77bf473e8daa8db462e815fb [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:57:01.435657 18606 raft_consensus.cc:2229] T 9263d97cbb774630a7c62957e1cdd77d P 0be3db2c50b7476da6339eff9de94b79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:57:01.436229 18605 tablet_replica.cc:330] T 9263d97cbb774630a7c62957e1cdd77d P e9a58779dcfa4b91b0e442d5c951c6e3: stopping tablet replica
I20260620 02:57:01.436239 18606 raft_consensus.cc:2258] T 9263d97cbb774630a7c62957e1cdd77d P 0be3db2c50b7476da6339eff9de94b79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:57:01.436688 18605 raft_consensus.cc:2229] T 9263d97cbb774630a7c62957e1cdd77d P e9a58779dcfa4b91b0e442d5c951c6e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:57:01.437186 18605 raft_consensus.cc:2258] T 9263d97cbb774630a7c62957e1cdd77d P e9a58779dcfa4b91b0e442d5c951c6e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:57:01.437793 18607 raft_consensus.cc:2258] T 9263d97cbb774630a7c62957e1cdd77d P 064c105c77bf473e8daa8db462e815fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:57:01.439440 18605 ts_tablet_manager.cc:1905] T 9263d97cbb774630a7c62957e1cdd77d P e9a58779dcfa4b91b0e442d5c951c6e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:57:01.439862 18607 ts_tablet_manager.cc:1905] T 9263d97cbb774630a7c62957e1cdd77d P 064c105c77bf473e8daa8db462e815fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:57:01.440375 18579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:01.424220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41310 (local address 127.0.97.1:39037)
0620 02:57:01.424392 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:01.424399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:01.429613 (+ 5214us) server_negotiation.cc:407] Connection header received
0620 02:57:01.430588 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:01.430597 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:01.430742 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:01.430835 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:01.432377 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.433691 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.435730 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.436417 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.436543 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:01.438676 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:01.438689 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:01.438704 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:01.439430 (+ 726us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:01.439492 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:01.439518 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:01.439605 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:01.439864 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 02:57:01.439973 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 02:57:01.441792 18297 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:57:01.441815 18606 ts_tablet_manager.cc:1905] T 9263d97cbb774630a7c62957e1cdd77d P 0be3db2c50b7476da6339eff9de94b79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:57:01.447970 18607 ts_tablet_manager.cc:1918] T 9263d97cbb774630a7c62957e1cdd77d P 064c105c77bf473e8daa8db462e815fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:57:01.448261 18607 log.cc:1196] T 9263d97cbb774630a7c62957e1cdd77d P 064c105c77bf473e8daa8db462e815fb: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-1/wal/wals/9263d97cbb774630a7c62957e1cdd77d
I20260620 02:57:01.448885 18607 ts_tablet_manager.cc:1939] T 9263d97cbb774630a7c62957e1cdd77d P 064c105c77bf473e8daa8db462e815fb: Deleting consensus metadata
I20260620 02:57:01.450244 18606 ts_tablet_manager.cc:1918] T 9263d97cbb774630a7c62957e1cdd77d P 0be3db2c50b7476da6339eff9de94b79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:57:01.450488 18606 log.cc:1196] T 9263d97cbb774630a7c62957e1cdd77d P 0be3db2c50b7476da6339eff9de94b79: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781923914418765-388-0/minicluster-data/ts-2/wal/wals/9263d97cbb774630a7c62957e1cdd77d
I20260620 02:57:01.450922 18101 catalog_manager.cc:4841] TS 064c105c77bf473e8daa8db462e815fb (127.0.97.2:40253): tablet 9263d97cbb774630a7c62957e1cdd77d (table test-table [id=4424d59c38244cc28c26bd452a8273ef]) successfully deleted
I20260620 02:57:01.451097 18606 ts_tablet_manager.cc:1939] T 9263d97cbb774630a7c62957e1cdd77d P 0be3db2c50b7476da6339eff9de94b79: Deleting consensus metadata
I20260620 02:57:01.453012 18102 catalog_manager.cc:4841] TS 0be3db2c50b7476da6339eff9de94b79 (127.0.97.3:46835): tablet 9263d97cbb774630a7c62957e1cdd77d (table test-table [id=4424d59c38244cc28c26bd452a8273ef]) successfully deleted
I20260620 02:57:01.534080 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 18175
I20260620 02:57:01.559172 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 18306
W20260620 02:57:01.559350 18100 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:39037 (error 108)
I20260620 02:57:01.562083 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:01.561656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39037 (local address 127.0.0.1:41318)
0620 02:57:01.561824 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:01.561938 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 02:57:01.562496 18100 catalog_manager.cc:4584] TS e9a58779dcfa4b91b0e442d5c951c6e3 (127.0.97.1:39037): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9263d97cbb774630a7c62957e1cdd77d: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39037: connect: Connection refused (error 111)
I20260620 02:57:01.568063 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:01.560037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48820 (local address 127.0.97.2:40253)
0620 02:57:01.560229 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:01.560235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:01.560444 (+ 209us) server_negotiation.cc:407] Connection header received
0620 02:57:01.560675 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:01.560681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:01.560807 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:01.560924 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:01.561617 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.562911 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.564278 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.564962 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.565098 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:01.566213 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:01.566224 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:01.566244 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:01.567181 (+ 937us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:01.567243 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:01.567269 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:01.567386 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:01.567553 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 02:57:01.567678 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 02:57:01.568849 18428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:57:01.616401 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:01.615881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39037 (local address 127.0.0.1:41332)
0620 02:57:01.616113 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:01.616222 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 02:57:01.617566 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:01.617170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39037 (local address 127.0.0.1:41344)
0620 02:57:01.617342 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:01.617428 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 02:57:01.678076 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:01.677554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39037 (local address 127.0.0.1:41346)
0620 02:57:01.677812 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:01.677897 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260620 02:57:01.679324 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:01.678873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39037 (local address 127.0.0.1:41358)
0620 02:57:01.679060 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:01.679153 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 02:57:01.680733 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 18306
I20260620 02:57:01.699324 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 18436
I20260620 02:57:01.706348 18580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:01.700087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49916 (local address 127.0.97.3:46835)
0620 02:57:01.700282 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:01.700288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:01.700386 (+ 98us) server_negotiation.cc:407] Connection header received
0620 02:57:01.700632 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:01.700636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:01.700717 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:01.700841 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:01.701239 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.701994 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.703357 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:01.703757 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:01.703894 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:01.705048 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:01.705057 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:01.705074 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:01.705659 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:01.705705 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:01.705724 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:01.705853 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:01.705948 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 02:57:01.706052 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 02:57:01.707161 18558 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:57:01.760128 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:01.759676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39037 (local address 127.0.0.1:41370)
0620 02:57:01.759884 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:01.759978 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 02:57:01.761197 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:01.760785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39037 (local address 127.0.0.1:41378)
0620 02:57:01.760993 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:01.761091 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 02:57:01.797240 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 18436
I20260620 02:57:01.813493 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 18085
I20260620 02:57:01.814752 18146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:57:01.928629 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 18085
2026-06-20T02:57:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2802 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:57:02 dist-test-slave-8p8c krb5kdc[18616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:57:02 dist-test-slave-8p8c krb5kdc[18616](info): set up 2 sockets
Jun 20 02:57:02 dist-test-slave-8p8c krb5kdc[18616](info): commencing operation
krb5kdc: starting...
W20260620 02:57:03.062618 388 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:57:03 dist-test-slave-8p8c krb5kdc[18616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:57:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:57:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:57:03.321735 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44281
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:57:03.650899 18629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:03.651262 18629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:03.651422 18629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:03.658972 18629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:57:03.659124 18629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:03.659247 18629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:03.659341 18629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:57:03.659430 18629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:57:03.669569 18629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:03.671041 18629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:03.672451 18629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:03.686335 18634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:03.686393 18635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:03.687224 18637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:03.688918 18629 server_base.cc:1034] running on GCE node
I20260620 02:57:03.689841 18629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:03.691399 18629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:03.692646 18629 hybrid_clock.cc:648] HybridClock initialized: now 1781924223692578 us; error 57 us; skew 500 ppm
I20260620 02:57:03.693090 18629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:57:03 dist-test-slave-8p8c krb5kdc[18616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:57:03.700488 18629 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:57:03.702265 18629 webserver.cc:460] Webserver started at http://127.0.97.62:42065/ using document root <none> and password file <none>
I20260620 02:57:03.703085 18629 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:03.703226 18629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:03.703586 18629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:03.706879 18629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "de9f09fcf5cd4bb6a168e217a6d7bdb9"
format_stamp: "Formatted at 2026-06-20 02:57:03 on dist-test-slave-8p8c"
server_key: "abb5f81bd438980cb6d1bd89ae78ec0d"
server_key_iv: "4537027e352aaae6c5868919883eca1a"
server_key_version: "encryptionkey@0"
I20260620 02:57:03.707710 18629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "de9f09fcf5cd4bb6a168e217a6d7bdb9"
format_stamp: "Formatted at 2026-06-20 02:57:03 on dist-test-slave-8p8c"
server_key: "abb5f81bd438980cb6d1bd89ae78ec0d"
server_key_iv: "4537027e352aaae6c5868919883eca1a"
server_key_version: "encryptionkey@0"
I20260620 02:57:03.713243 18629 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 02:57:03.717325 18644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:03.718999 18629 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260620 02:57:03.719197 18629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "de9f09fcf5cd4bb6a168e217a6d7bdb9"
format_stamp: "Formatted at 2026-06-20 02:57:03 on dist-test-slave-8p8c"
server_key: "abb5f81bd438980cb6d1bd89ae78ec0d"
server_key_iv: "4537027e352aaae6c5868919883eca1a"
server_key_version: "encryptionkey@0"
I20260620 02:57:03.719424 18629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:03.735023 18629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:03.739759 18629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:03.740065 18629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:03.759743 18629 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44281
I20260620 02:57:03.759778 18695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44281 every 8 connection(s)
I20260620 02:57:03.761165 18629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:57:03.765002 18696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:03.765858 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 18629
I20260620 02:57:03.766088 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:57:03.766743 388 external_mini_cluster.cc:1428] Setting key 819fd231fe12b2269cfb97a38452c627
I20260620 02:57:03.779879 18696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de9f09fcf5cd4bb6a168e217a6d7bdb9: Bootstrap starting.
Jun 20 02:57:03 dist-test-slave-8p8c krb5kdc[18616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:57:03.783165 18696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de9f09fcf5cd4bb6a168e217a6d7bdb9: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:03.784288 18696 log.cc:824] T 00000000000000000000000000000000 P de9f09fcf5cd4bb6a168e217a6d7bdb9: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:03.787346 18696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de9f09fcf5cd4bb6a168e217a6d7bdb9: No bootstrap required, opened a new log
I20260620 02:57:03.791460 18699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:03.768676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46818 (local address 127.0.97.62:44281)
0620 02:57:03.769502 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:03.769513 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:03.769556 (+ 43us) server_negotiation.cc:407] Connection header received
0620 02:57:03.771542 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:03.771569 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:03.771954 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:03.772315 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:03.774071 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:03.776045 (+ 1974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:03.778014 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:03.778783 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:03.778977 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:03.783392 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:03.783402 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:03.783417 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:03.783462 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:03.786893 (+ 3431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:03.787592 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:03.787600 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:03.787607 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:03.787827 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:03.788234 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:03.788239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:03.788244 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:03.788902 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:03.789221 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:03.789565 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 02:57:03.790043 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":423,"threads_started":1}
I20260620 02:57:03.796939 18696 raft_consensus.cc:348] T 00000000000000000000000000000000 P de9f09fcf5cd4bb6a168e217a6d7bdb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9f09fcf5cd4bb6a168e217a6d7bdb9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44281 } }
I20260620 02:57:03.797314 18696 raft_consensus.cc:374] T 00000000000000000000000000000000 P de9f09fcf5cd4bb6a168e217a6d7bdb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:03.797385 18696 raft_consensus.cc:729] T 00000000000000000000000000000000 P de9f09fcf5cd4bb6a168e217a6d7bdb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de9f09fcf5cd4bb6a168e217a6d7bdb9, State: Initialized, Role: FOLLOWER
I20260620 02:57:03.797940 18696 consensus_queue.cc:260] T 00000000000000000000000000000000 P de9f09fcf5cd4bb6a168e217a6d7bdb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9f09fcf5cd4bb6a168e217a6d7bdb9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44281 } }
I20260620 02:57:03.798146 18696 raft_consensus.cc:388] T 00000000000000000000000000000000 P de9f09fcf5cd4bb6a168e217a6d7bdb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:57:03.798235 18696 raft_consensus.cc:482] T 00000000000000000000000000000000 P de9f09fcf5cd4bb6a168e217a6d7bdb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:57:03.798341 18696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de9f09fcf5cd4bb6a168e217a6d7bdb9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:03.800521 18696 raft_consensus.cc:504] T 00000000000000000000000000000000 P de9f09fcf5cd4bb6a168e217a6d7bdb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9f09fcf5cd4bb6a168e217a6d7bdb9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44281 } }
I20260620 02:57:03.800966 18696 leader_election.cc:302] T 00000000000000000000000000000000 P de9f09fcf5cd4bb6a168e217a6d7bdb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de9f09fcf5cd4bb6a168e217a6d7bdb9; no voters:
I20260620 02:57:03.801554 18696 leader_election.cc:288] T 00000000000000000000000000000000 P de9f09fcf5cd4bb6a168e217a6d7bdb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:57:03.801708 18701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de9f09fcf5cd4bb6a168e217a6d7bdb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:57:03.803215 18701 raft_consensus.cc:686] T 00000000000000000000000000000000 P de9f09fcf5cd4bb6a168e217a6d7bdb9 [term 1 LEADER]: Becoming Leader. State: Replica: de9f09fcf5cd4bb6a168e217a6d7bdb9, State: Running, Role: LEADER
I20260620 02:57:03.803813 18701 consensus_queue.cc:237] T 00000000000000000000000000000000 P de9f09fcf5cd4bb6a168e217a6d7bdb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9f09fcf5cd4bb6a168e217a6d7bdb9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44281 } }
I20260620 02:57:03.804538 18696 sys_catalog.cc:564] T 00000000000000000000000000000000 P de9f09fcf5cd4bb6a168e217a6d7bdb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:57:03.806746 18703 sys_catalog.cc:455] T 00000000000000000000000000000000 P de9f09fcf5cd4bb6a168e217a6d7bdb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de9f09fcf5cd4bb6a168e217a6d7bdb9. Latest consensus state: current_term: 1 leader_uuid: "de9f09fcf5cd4bb6a168e217a6d7bdb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9f09fcf5cd4bb6a168e217a6d7bdb9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44281 } } }
I20260620 02:57:03.807119 18703 sys_catalog.cc:458] T 00000000000000000000000000000000 P de9f09fcf5cd4bb6a168e217a6d7bdb9 [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:03.807811 18702 sys_catalog.cc:455] T 00000000000000000000000000000000 P de9f09fcf5cd4bb6a168e217a6d7bdb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de9f09fcf5cd4bb6a168e217a6d7bdb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9f09fcf5cd4bb6a168e217a6d7bdb9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44281 } } }
I20260620 02:57:03.808161 18702 sys_catalog.cc:458] T 00000000000000000000000000000000 P de9f09fcf5cd4bb6a168e217a6d7bdb9 [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:03.808889 18708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:57:03.815042 18708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:57:03.824023 18708 catalog_manager.cc:1347] Generated new cluster ID: 999395d032204d769aba9ad1d89c23e0
I20260620 02:57:03.824193 18708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:57:03.839504 18708 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:57:03.840901 18708 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:57:03.846853 18708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de9f09fcf5cd4bb6a168e217a6d7bdb9: Generated new TSK 0
I20260620 02:57:03.848112 18708 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:57:03.922962 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44281
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:57:04.287189 18722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:04.287482 18722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:04.287636 18722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:04.295238 18722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:04.295348 18722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:04.295518 18722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:57:04.305773 18722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:04.307186 18722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:04.308571 18722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:04.322705 18727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:04.322739 18728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:04.322705 18730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:04.323829 18722 server_base.cc:1034] running on GCE node
I20260620 02:57:04.324443 18722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:04.325488 18722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:04.326713 18722 hybrid_clock.cc:648] HybridClock initialized: now 1781924224326660 us; error 45 us; skew 500 ppm
I20260620 02:57:04.327044 18722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:57:04 dist-test-slave-8p8c krb5kdc[18616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:57:04.330308 18722 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:57:04.331784 18722 webserver.cc:460] Webserver started at http://127.0.97.1:42537/ using document root <none> and password file <none>
I20260620 02:57:04.332453 18722 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:04.332551 18722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:04.332798 18722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:04.335808 18722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-0/data/instance:
uuid: "2ca62222b98e4ba5b15a97d37ca685c4"
format_stamp: "Formatted at 2026-06-20 02:57:04 on dist-test-slave-8p8c"
server_key: "e48407791757e3252fdafd33f230c1cb"
server_key_iv: "1443e47a0c169f4a3ac6bf9f14651f23"
server_key_version: "encryptionkey@0"
I20260620 02:57:04.336575 18722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2ca62222b98e4ba5b15a97d37ca685c4"
format_stamp: "Formatted at 2026-06-20 02:57:04 on dist-test-slave-8p8c"
server_key: "e48407791757e3252fdafd33f230c1cb"
server_key_iv: "1443e47a0c169f4a3ac6bf9f14651f23"
server_key_version: "encryptionkey@0"
I20260620 02:57:04.341629 18722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 02:57:04.345309 18737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:04.347084 18722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:57:04.347257 18722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-0/wal
uuid: "2ca62222b98e4ba5b15a97d37ca685c4"
format_stamp: "Formatted at 2026-06-20 02:57:04 on dist-test-slave-8p8c"
server_key: "e48407791757e3252fdafd33f230c1cb"
server_key_iv: "1443e47a0c169f4a3ac6bf9f14651f23"
server_key_version: "encryptionkey@0"
I20260620 02:57:04.347469 18722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:04.364311 18722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:04.368968 18722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:04.369323 18722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:04.370337 18722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:57:04.371889 18722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:57:04.371961 18722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:04.372102 18722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:57:04.372170 18722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:04.407389 18722 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36093
I20260620 02:57:04.407472 18849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36093 every 8 connection(s)
I20260620 02:57:04.408720 18722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:57:04.412326 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 18722
I20260620 02:57:04.412561 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:57:04.413203 388 external_mini_cluster.cc:1428] Setting key ceae2d533d7dc90f05f0d719d81aebe1
Jun 20 02:57:04 dist-test-slave-8p8c krb5kdc[18616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:57:04.434724 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:04.412269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44281 (local address 127.0.97.1:55795)
0620 02:57:04.413203 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:04.413242 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:04.415554 (+ 2312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:04.416278 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:04.416287 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:04.416811 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:57:04.417424 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:04.417444 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:04.418552 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:04.418556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:04.419809 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:04.419821 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:04.420624 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:04.420634 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:04.420894 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:04.421660 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:57:04.421684 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:57:04.424388 (+ 2704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:57:04.429299 (+ 4911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:57:04.429309 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:57:04.429364 (+ 55us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:57:04.430095 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:57:04.430600 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:57:04.430642 (+ 42us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:57:04.430648 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:57:04.430943 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:57:04.431903 (+ 960us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:57:04.431912 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:57:04.432454 (+ 542us) client_negotiation.cc:769] Sending connection context
0620 02:57:04.432854 (+ 400us) client_negotiation.cc:240] Negotiation successful
0620 02:57:04.433460 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":420,"threads_started":1}
I20260620 02:57:04.435211 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:04.411615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55795 (local address 127.0.97.62:44281)
0620 02:57:04.412135 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:04.412139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:04.413387 (+ 1248us) server_negotiation.cc:407] Connection header received
0620 02:57:04.415818 (+ 2431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:04.415823 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:04.415907 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:04.416028 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:04.417595 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:04.418321 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:04.419957 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:04.420363 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:04.420513 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:04.424710 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:04.424720 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:04.424728 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:04.424777 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:04.429115 (+ 4338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:04.430250 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:04.430255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:04.430259 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:04.430452 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:04.431093 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:04.431100 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:04.431142 (+ 42us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:04.431638 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:04.431788 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:04.434589 (+ 2801us) server_negotiation.cc:299] Negotiation successful
0620 02:57:04.434852 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":300,"threads_started":1}
I20260620 02:57:04.436807 18850 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44281
I20260620 02:57:04.437201 18850 heartbeater.cc:464] Registering TS with master...
I20260620 02:57:04.438203 18850 heartbeater.cc:511] Master 127.0.97.62:44281 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 02:57:04.440626 18661 ts_manager.cc:194] Registered new tserver with Master: 2ca62222b98e4ba5b15a97d37ca685c4 (127.0.97.1:36093)
I20260620 02:57:04.442168 18661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:57:04.474480 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44281
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:57:04.797967 18856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:04.798300 18856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:04.798485 18856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:04.805804 18856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:04.805946 18856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:04.806121 18856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:57:04.815927 18856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:04.817198 18856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:04.818531 18856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:04.831856 18861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:04.831928 18862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:04.831959 18864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:04.832574 18856 server_base.cc:1034] running on GCE node
I20260620 02:57:04.833125 18856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:04.833967 18856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:04.835160 18856 hybrid_clock.cc:648] HybridClock initialized: now 1781924224835106 us; error 42 us; skew 500 ppm
I20260620 02:57:04.835428 18856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:57:04 dist-test-slave-8p8c krb5kdc[18616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:57:04.838497 18856 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:57:04.839941 18856 webserver.cc:460] Webserver started at http://127.0.97.2:42257/ using document root <none> and password file <none>
I20260620 02:57:04.840631 18856 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:04.840751 18856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:04.841063 18856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:04.843891 18856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-1/data/instance:
uuid: "3606b16577d7430eb91c26161e8d8d11"
format_stamp: "Formatted at 2026-06-20 02:57:04 on dist-test-slave-8p8c"
server_key: "9e96508e442b2336bfea944b4059a9f4"
server_key_iv: "9fbf1fb6a64d3e08e42becb2304fda99"
server_key_version: "encryptionkey@0"
I20260620 02:57:04.844671 18856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3606b16577d7430eb91c26161e8d8d11"
format_stamp: "Formatted at 2026-06-20 02:57:04 on dist-test-slave-8p8c"
server_key: "9e96508e442b2336bfea944b4059a9f4"
server_key_iv: "9fbf1fb6a64d3e08e42becb2304fda99"
server_key_version: "encryptionkey@0"
I20260620 02:57:04.849499 18856 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 02:57:04.853204 18871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:04.854786 18856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:57:04.854950 18856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-1/wal
uuid: "3606b16577d7430eb91c26161e8d8d11"
format_stamp: "Formatted at 2026-06-20 02:57:04 on dist-test-slave-8p8c"
server_key: "9e96508e442b2336bfea944b4059a9f4"
server_key_iv: "9fbf1fb6a64d3e08e42becb2304fda99"
server_key_version: "encryptionkey@0"
I20260620 02:57:04.855156 18856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:04.883459 18856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:04.888314 18856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:04.888660 18856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:04.889742 18856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:57:04.891312 18856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:57:04.891390 18856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:04.891548 18856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:57:04.891605 18856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:04.927922 18856 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44375
I20260620 02:57:04.928007 18983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44375 every 8 connection(s)
I20260620 02:57:04.929203 18856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:57:04.938637 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 18856
I20260620 02:57:04.938838 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:57:04.939260 388 external_mini_cluster.cc:1428] Setting key b4bc7aa46e01091c95c0be616a7383de
Jun 20 02:57:04 dist-test-slave-8p8c krb5kdc[18616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:57:04.949756 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:04.931692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57199 (local address 127.0.97.62:44281)
0620 02:57:04.931864 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:04.931868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:04.933127 (+ 1259us) server_negotiation.cc:407] Connection header received
0620 02:57:04.935027 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:04.935032 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:04.935111 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:04.935205 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:04.936996 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:04.937758 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:04.939642 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:04.940087 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:04.940232 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:04.943805 (+ 3573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:04.943811 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:04.943814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:04.943848 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:04.946671 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:04.947441 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:04.947446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:04.947448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:04.947594 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:04.948048 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:04.948052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:04.948054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:04.948320 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:04.948476 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:04.949179 (+ 703us) server_negotiation.cc:299] Negotiation successful
0620 02:57:04.949414 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 02:57:04.950882 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:04.932134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44281 (local address 127.0.97.2:57199)
0620 02:57:04.932926 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:04.932967 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:04.934705 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:04.935451 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:04.935459 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:04.936204 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:57:04.936802 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:04.936821 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:04.937911 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:04.937915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:04.939493 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:04.939505 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:04.940317 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:04.940324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:04.940615 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:04.941378 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:57:04.941401 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:57:04.943560 (+ 2159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:57:04.946820 (+ 3260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:57:04.946830 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:57:04.946849 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:57:04.947297 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:57:04.947707 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:57:04.947712 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:57:04.947716 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:57:04.947909 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:57:04.948449 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:57:04.948457 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:57:04.948811 (+ 354us) client_negotiation.cc:769] Sending connection context
0620 02:57:04.949076 (+ 265us) client_negotiation.cc:240] Negotiation successful
0620 02:57:04.949634 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":372,"threads_started":1}
I20260620 02:57:04.952396 18984 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44281
I20260620 02:57:04.952723 18984 heartbeater.cc:464] Registering TS with master...
I20260620 02:57:04.953580 18984 heartbeater.cc:511] Master 127.0.97.62:44281 requested a full tablet report, sending...
I20260620 02:57:04.955430 18661 ts_manager.cc:194] Registered new tserver with Master: 3606b16577d7430eb91c26161e8d8d11 (127.0.97.2:44375)
I20260620 02:57:04.956710 18661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57199
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:57:04.999187 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44281
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:57:05.324558 18989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:05.324851 18989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:05.324996 18989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:05.332378 18989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:05.332489 18989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:05.332681 18989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:57:05.342646 18989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:05.343927 18989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:05.345219 18989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:05.359066 18994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:05.359113 18995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:05.359148 18997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:05.360828 18989 server_base.cc:1034] running on GCE node
I20260620 02:57:05.361635 18989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:05.362821 18989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:05.364089 18989 hybrid_clock.cc:648] HybridClock initialized: now 1781924225364043 us; error 64 us; skew 500 ppm
I20260620 02:57:05.364446 18989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:57:05 dist-test-slave-8p8c krb5kdc[18616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:57:05.367903 18989 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:57:05.369422 18989 webserver.cc:460] Webserver started at http://127.0.97.3:35765/ using document root <none> and password file <none>
I20260620 02:57:05.370100 18989 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:05.370198 18989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:05.370525 18989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:05.373404 18989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-2/data/instance:
uuid: "78eb0eb4842642638f7dbe2f7041343f"
format_stamp: "Formatted at 2026-06-20 02:57:05 on dist-test-slave-8p8c"
server_key: "33b737797e910281abe2b0e142e4d3a9"
server_key_iv: "7d80ab208674458c8e3d9e18270ce33e"
server_key_version: "encryptionkey@0"
I20260620 02:57:05.374140 18989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-2/wal/instance:
uuid: "78eb0eb4842642638f7dbe2f7041343f"
format_stamp: "Formatted at 2026-06-20 02:57:05 on dist-test-slave-8p8c"
server_key: "33b737797e910281abe2b0e142e4d3a9"
server_key_iv: "7d80ab208674458c8e3d9e18270ce33e"
server_key_version: "encryptionkey@0"
I20260620 02:57:05.379424 18989 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 02:57:05.382970 19004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:05.384326 18989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:57:05.384496 18989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-2/wal
uuid: "78eb0eb4842642638f7dbe2f7041343f"
format_stamp: "Formatted at 2026-06-20 02:57:05 on dist-test-slave-8p8c"
server_key: "33b737797e910281abe2b0e142e4d3a9"
server_key_iv: "7d80ab208674458c8e3d9e18270ce33e"
server_key_version: "encryptionkey@0"
I20260620 02:57:05.384709 18989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:05.399056 18989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:05.403784 18989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:05.404103 18989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:05.405234 18989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:57:05.406719 18989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:57:05.406814 18989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:05.406968 18989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:57:05.407033 18989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:05.441932 18989 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46153
I20260620 02:57:05.441980 19116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46153 every 8 connection(s)
I20260620 02:57:05.443354 18989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:57:05.443624 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 18989
I20260620 02:57:05.443799 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:57:05.444453 388 external_mini_cluster.cc:1428] Setting key 199d1d5354bb28ab81c89acb68cef983
I20260620 02:57:05.445392 18850 heartbeater.cc:503] Master 127.0.97.62:44281 was elected leader, sending a full tablet report...
Jun 20 02:57:05 dist-test-slave-8p8c krb5kdc[18616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:57:05.464679 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:05.446383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34201 (local address 127.0.97.62:44281)
0620 02:57:05.446570 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:05.446576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:05.448071 (+ 1495us) server_negotiation.cc:407] Connection header received
0620 02:57:05.449844 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:05.449849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:05.449930 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:05.450046 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:05.451627 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:05.452470 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:05.453992 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:05.454449 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:05.454580 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:05.458636 (+ 4056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:05.458643 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:05.458650 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:05.458691 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:05.461656 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:05.462422 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:05.462435 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:05.462438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:05.462563 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:05.463000 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:05.463003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:05.463006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:05.463266 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:05.463420 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:05.464127 (+ 707us) server_negotiation.cc:299] Negotiation successful
0620 02:57:05.464346 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 02:57:05.465598 19119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:05.447044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44281 (local address 127.0.97.3:34201)
0620 02:57:05.447912 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:05.447965 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:05.449593 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:05.450282 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:05.450291 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:05.450915 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:57:05.451458 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:05.451474 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:05.452632 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:05.452637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:05.453845 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:05.453856 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:05.454680 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:05.454686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:05.454919 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:05.455590 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:57:05.455617 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:57:05.458267 (+ 2650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:57:05.461796 (+ 3529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:57:05.461805 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:57:05.461822 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:57:05.462243 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:57:05.462675 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:57:05.462680 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:57:05.462684 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:57:05.462870 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:57:05.463394 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:57:05.463401 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:57:05.463751 (+ 350us) client_negotiation.cc:769] Sending connection context
0620 02:57:05.464001 (+ 250us) client_negotiation.cc:240] Negotiation successful
0620 02:57:05.464516 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":281,"threads_started":1}
I20260620 02:57:05.467011 19117 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44281
I20260620 02:57:05.467339 19117 heartbeater.cc:464] Registering TS with master...
I20260620 02:57:05.468097 19117 heartbeater.cc:511] Master 127.0.97.62:44281 requested a full tablet report, sending...
I20260620 02:57:05.469512 18660 ts_manager.cc:194] Registered new tserver with Master: 78eb0eb4842642638f7dbe2f7041343f (127.0.97.3:46153)
I20260620 02:57:05.470518 18660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34201
I20260620 02:57:05.473050 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:57:05.490753 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:05.477411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46832 (local address 127.0.97.62:44281)
0620 02:57:05.477586 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:05.477590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:05.478025 (+ 435us) server_negotiation.cc:407] Connection header received
0620 02:57:05.478173 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:05.478177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:05.478254 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:05.478341 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:05.479720 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:05.480393 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:05.481764 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:05.482192 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:05.482294 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:05.485222 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:05.485227 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:05.485231 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:05.485262 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:05.488019 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:05.488854 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:05.488858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:05.488860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:05.488972 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:05.489446 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:05.489449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:05.489451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:05.489797 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:05.490001 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:05.490328 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 02:57:05.490522 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:57:05.495366 18660 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:57:05.502866 18660 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:57:05.527946 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:05.516404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44375 (local address 127.0.0.1:44610)
0620 02:57:05.518559 (+ 2155us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:05.518615 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:05.518811 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:05.520003 (+ 1192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:05.520011 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:05.520046 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:57:05.520768 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:05.520789 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:05.522640 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:05.522646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:05.525050 (+ 2404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:05.525063 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:05.526973 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:05.526980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:05.527185 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:05.527239 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 02:57:05.527382 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 02:57:05.527580 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1849,"thread_start_us":1695,"threads_started":1}
I20260620 02:57:05.528899 19131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:05.516637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44610 (local address 127.0.97.2:44375)
0620 02:57:05.518051 (+ 1414us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:05.518064 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:05.519020 (+ 956us) server_negotiation.cc:407] Connection header received
0620 02:57:05.519140 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:05.519152 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:05.519448 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:05.519682 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:57:05.520966 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:05.522497 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:05.525215 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:05.526800 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:05.526933 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:05.527153 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:05.527436 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 02:57:05.527625 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1199,"thread_start_us":1056,"threads_started":1}
I20260620 02:57:05.529521 19134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:05.519844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46153 (local address 127.0.0.1:44834)
0620 02:57:05.521348 (+ 1504us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:05.521361 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:05.521493 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:05.522118 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:05.522124 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:05.522151 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:57:05.522512 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:05.522533 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:05.524279 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:05.524284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:05.526591 (+ 2307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:05.526606 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:05.528751 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:05.528761 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:05.528934 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:05.528964 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 02:57:05.529047 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 02:57:05.529176 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1395,"mutex_wait_us":57,"thread_start_us":1275,"threads_started":1}
I20260620 02:57:05.529726 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:05.516569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36093 (local address 127.0.0.1:53926)
0620 02:57:05.520733 (+ 4164us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:05.520753 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:05.520875 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:05.521566 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:05.521571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:05.521595 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:57:05.521952 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:05.521965 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:05.525086 (+ 3121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:05.525091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:05.526626 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:05.526638 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:05.528813 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:05.528819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:05.529023 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:05.529062 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 02:57:05.529151 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 02:57:05.529301 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4001,"mutex_wait_us":23,"thread_start_us":2902,"threads_started":1}
I20260620 02:57:05.530315 19133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:05.519386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44834 (local address 127.0.97.3:46153)
0620 02:57:05.520224 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:05.520235 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:05.521454 (+ 1219us) server_negotiation.cc:407] Connection header received
0620 02:57:05.521687 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:05.521695 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:05.521912 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:05.522113 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:57:05.522703 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:05.524111 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:05.526778 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:05.528243 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:05.528372 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:05.528621 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:05.529786 (+ 1165us) server_negotiation.cc:299] Negotiation successful
0620 02:57:05.529964 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":562,"threads_started":1}
I20260620 02:57:05.530510 19132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:05.517089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53926 (local address 127.0.97.1:36093)
0620 02:57:05.519302 (+ 2213us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:05.519314 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:05.521008 (+ 1694us) server_negotiation.cc:407] Connection header received
0620 02:57:05.521120 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:05.521129 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:05.521380 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:05.521588 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:57:05.522171 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:05.523858 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:05.526831 (+ 2973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:05.528286 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:05.528419 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:05.528653 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:05.530014 (+ 1361us) server_negotiation.cc:299] Negotiation successful
0620 02:57:05.530187 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2022,"thread_start_us":1870,"threads_started":1}
I20260620 02:57:05.533322 18919 tablet_service.cc:1467] Processing CreateTablet for tablet c15a6867c87b4feb9dd33fe3f19e63fe (DEFAULT_TABLE table=test-table [id=dff0ac55569c4f398a73bcab37e08980]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:57:05.533572 19052 tablet_service.cc:1467] Processing CreateTablet for tablet c15a6867c87b4feb9dd33fe3f19e63fe (DEFAULT_TABLE table=test-table [id=dff0ac55569c4f398a73bcab37e08980]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:57:05.534091 18785 tablet_service.cc:1467] Processing CreateTablet for tablet c15a6867c87b4feb9dd33fe3f19e63fe (DEFAULT_TABLE table=test-table [id=dff0ac55569c4f398a73bcab37e08980]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:57:05.534936 18919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c15a6867c87b4feb9dd33fe3f19e63fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:05.534961 19052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c15a6867c87b4feb9dd33fe3f19e63fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:05.535362 18785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c15a6867c87b4feb9dd33fe3f19e63fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:05.548643 19135 tablet_bootstrap.cc:492] T c15a6867c87b4feb9dd33fe3f19e63fe P 78eb0eb4842642638f7dbe2f7041343f: Bootstrap starting.
I20260620 02:57:05.550253 19136 tablet_bootstrap.cc:492] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11: Bootstrap starting.
I20260620 02:57:05.551046 19137 tablet_bootstrap.cc:492] T c15a6867c87b4feb9dd33fe3f19e63fe P 2ca62222b98e4ba5b15a97d37ca685c4: Bootstrap starting.
I20260620 02:57:05.552637 19135 tablet_bootstrap.cc:654] T c15a6867c87b4feb9dd33fe3f19e63fe P 78eb0eb4842642638f7dbe2f7041343f: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:05.553989 19135 log.cc:824] T c15a6867c87b4feb9dd33fe3f19e63fe P 78eb0eb4842642638f7dbe2f7041343f: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:05.554497 19136 tablet_bootstrap.cc:654] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:05.554740 19137 tablet_bootstrap.cc:654] T c15a6867c87b4feb9dd33fe3f19e63fe P 2ca62222b98e4ba5b15a97d37ca685c4: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:05.555557 19136 log.cc:824] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:05.556340 19137 log.cc:824] T c15a6867c87b4feb9dd33fe3f19e63fe P 2ca62222b98e4ba5b15a97d37ca685c4: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:05.563163 19137 tablet_bootstrap.cc:492] T c15a6867c87b4feb9dd33fe3f19e63fe P 2ca62222b98e4ba5b15a97d37ca685c4: No bootstrap required, opened a new log
I20260620 02:57:05.563582 19137 ts_tablet_manager.cc:1397] T c15a6867c87b4feb9dd33fe3f19e63fe P 2ca62222b98e4ba5b15a97d37ca685c4: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260620 02:57:05.563723 19135 tablet_bootstrap.cc:492] T c15a6867c87b4feb9dd33fe3f19e63fe P 78eb0eb4842642638f7dbe2f7041343f: No bootstrap required, opened a new log
I20260620 02:57:05.563673 19136 tablet_bootstrap.cc:492] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11: No bootstrap required, opened a new log
I20260620 02:57:05.563998 19135 ts_tablet_manager.cc:1397] T c15a6867c87b4feb9dd33fe3f19e63fe P 78eb0eb4842642638f7dbe2f7041343f: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.003s
I20260620 02:57:05.564205 19136 ts_tablet_manager.cc:1397] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.009s
I20260620 02:57:05.572108 19136 raft_consensus.cc:348] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3606b16577d7430eb91c26161e8d8d11" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44375 } } peers { permanent_uuid: "2ca62222b98e4ba5b15a97d37ca685c4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36093 } } peers { permanent_uuid: "78eb0eb4842642638f7dbe2f7041343f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46153 } }
I20260620 02:57:05.572733 19136 raft_consensus.cc:374] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:05.572865 19136 raft_consensus.cc:729] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3606b16577d7430eb91c26161e8d8d11, State: Initialized, Role: FOLLOWER
I20260620 02:57:05.573547 19136 consensus_queue.cc:260] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3606b16577d7430eb91c26161e8d8d11" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44375 } } peers { permanent_uuid: "2ca62222b98e4ba5b15a97d37ca685c4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36093 } } peers { permanent_uuid: "78eb0eb4842642638f7dbe2f7041343f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46153 } }
I20260620 02:57:05.574764 18984 heartbeater.cc:503] Master 127.0.97.62:44281 was elected leader, sending a full tablet report...
I20260620 02:57:05.574770 19135 raft_consensus.cc:348] T c15a6867c87b4feb9dd33fe3f19e63fe P 78eb0eb4842642638f7dbe2f7041343f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3606b16577d7430eb91c26161e8d8d11" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44375 } } peers { permanent_uuid: "2ca62222b98e4ba5b15a97d37ca685c4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36093 } } peers { permanent_uuid: "78eb0eb4842642638f7dbe2f7041343f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46153 } }
I20260620 02:57:05.574774 19137 raft_consensus.cc:348] T c15a6867c87b4feb9dd33fe3f19e63fe P 2ca62222b98e4ba5b15a97d37ca685c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3606b16577d7430eb91c26161e8d8d11" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44375 } } peers { permanent_uuid: "2ca62222b98e4ba5b15a97d37ca685c4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36093 } } peers { permanent_uuid: "78eb0eb4842642638f7dbe2f7041343f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46153 } }
I20260620 02:57:05.575244 19137 raft_consensus.cc:374] T c15a6867c87b4feb9dd33fe3f19e63fe P 2ca62222b98e4ba5b15a97d37ca685c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:05.575246 19135 raft_consensus.cc:374] T c15a6867c87b4feb9dd33fe3f19e63fe P 78eb0eb4842642638f7dbe2f7041343f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:05.575349 19135 raft_consensus.cc:729] T c15a6867c87b4feb9dd33fe3f19e63fe P 78eb0eb4842642638f7dbe2f7041343f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78eb0eb4842642638f7dbe2f7041343f, State: Initialized, Role: FOLLOWER
I20260620 02:57:05.575346 19137 raft_consensus.cc:729] T c15a6867c87b4feb9dd33fe3f19e63fe P 2ca62222b98e4ba5b15a97d37ca685c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca62222b98e4ba5b15a97d37ca685c4, State: Initialized, Role: FOLLOWER
I20260620 02:57:05.575987 19135 consensus_queue.cc:260] T c15a6867c87b4feb9dd33fe3f19e63fe P 78eb0eb4842642638f7dbe2f7041343f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3606b16577d7430eb91c26161e8d8d11" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44375 } } peers { permanent_uuid: "2ca62222b98e4ba5b15a97d37ca685c4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36093 } } peers { permanent_uuid: "78eb0eb4842642638f7dbe2f7041343f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46153 } }
I20260620 02:57:05.575990 19137 consensus_queue.cc:260] T c15a6867c87b4feb9dd33fe3f19e63fe P 2ca62222b98e4ba5b15a97d37ca685c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3606b16577d7430eb91c26161e8d8d11" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44375 } } peers { permanent_uuid: "2ca62222b98e4ba5b15a97d37ca685c4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36093 } } peers { permanent_uuid: "78eb0eb4842642638f7dbe2f7041343f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46153 } }
I20260620 02:57:05.578088 19137 ts_tablet_manager.cc:1428] T c15a6867c87b4feb9dd33fe3f19e63fe P 2ca62222b98e4ba5b15a97d37ca685c4: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260620 02:57:05.579774 19136 ts_tablet_manager.cc:1428] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11: Time spent starting tablet: real 0.015s user 0.010s sys 0.000s
I20260620 02:57:05.583232 19117 heartbeater.cc:503] Master 127.0.97.62:44281 was elected leader, sending a full tablet report...
I20260620 02:57:05.584101 19135 ts_tablet_manager.cc:1428] T c15a6867c87b4feb9dd33fe3f19e63fe P 78eb0eb4842642638f7dbe2f7041343f: Time spent starting tablet: real 0.020s user 0.013s sys 0.007s
W20260620 02:57:05.661247 18851 tablet.cc:2369] T c15a6867c87b4feb9dd33fe3f19e63fe P 2ca62222b98e4ba5b15a97d37ca685c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:57:05.681341 18985 tablet.cc:2369] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:57:05.695182 19118 tablet.cc:2369] T c15a6867c87b4feb9dd33fe3f19e63fe P 78eb0eb4842642638f7dbe2f7041343f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:57:05.948278 19141 raft_consensus.cc:482] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:57:05.948678 19141 raft_consensus.cc:504] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3606b16577d7430eb91c26161e8d8d11" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44375 } } peers { permanent_uuid: "2ca62222b98e4ba5b15a97d37ca685c4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36093 } } peers { permanent_uuid: "78eb0eb4842642638f7dbe2f7041343f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46153 } }
I20260620 02:57:05.950626 19141 leader_election.cc:288] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ca62222b98e4ba5b15a97d37ca685c4 (127.0.97.1:36093), 78eb0eb4842642638f7dbe2f7041343f (127.0.97.3:46153)
I20260620 02:57:05.960443 19133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:05.951187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43717 (local address 127.0.97.3:46153)
0620 02:57:05.951416 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:05.951424 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:05.951990 (+ 566us) server_negotiation.cc:407] Connection header received
0620 02:57:05.952354 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:05.952359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:05.952444 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:05.952558 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:57:05.953450 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:05.954682 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:05.957954 (+ 3272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:05.959258 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:05.959396 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:05.959576 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:05.960152 (+ 576us) server_negotiation.cc:299] Negotiation successful
0620 02:57:05.960255 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 02:57:05.960546 19132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:05.951203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38267 (local address 127.0.97.1:36093)
0620 02:57:05.951474 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:05.951479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:05.952083 (+ 604us) server_negotiation.cc:407] Connection header received
0620 02:57:05.952352 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:05.952358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:05.952479 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:05.952569 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:57:05.953450 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:05.954658 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:05.957948 (+ 3290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:05.959364 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:05.959486 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:05.959672 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:05.960167 (+ 495us) server_negotiation.cc:299] Negotiation successful
0620 02:57:05.960305 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 02:57:05.960630 19144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:05.950918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46153 (local address 127.0.97.2:43717)
0620 02:57:05.951832 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:05.951859 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:05.952051 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:05.952644 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:05.952650 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:05.952684 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:57:05.953202 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:05.953225 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:05.954837 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:05.954843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:05.957742 (+ 2899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:05.957759 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:05.959588 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:05.959597 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:05.959873 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:05.959946 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 02:57:05.960066 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 02:57:05.960214 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":269,"thread_start_us":482,"threads_started":1}
I20260620 02:57:05.961179 19145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:05.950981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36093 (local address 127.0.97.2:38267)
0620 02:57:05.951969 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:05.951986 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:05.952090 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:05.952644 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:05.952650 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:05.952752 (+ 102us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:57:05.953261 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:05.953273 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:05.954836 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:05.954843 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:05.957742 (+ 2899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:05.957759 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:05.959682 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:05.959689 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:05.959923 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:05.959946 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 02:57:05.960743 (+ 797us) client_negotiation.cc:240] Negotiation successful
0620 02:57:05.960886 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"mutex_wait_us":257,"spinlock_wait_cycles":23424,"thread_start_us":672,"threads_started":1}
I20260620 02:57:05.961820 19072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15a6867c87b4feb9dd33fe3f19e63fe" candidate_uuid: "3606b16577d7430eb91c26161e8d8d11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78eb0eb4842642638f7dbe2f7041343f" is_pre_election: true
I20260620 02:57:05.962438 19072 raft_consensus.cc:2454] T c15a6867c87b4feb9dd33fe3f19e63fe P 78eb0eb4842642638f7dbe2f7041343f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3606b16577d7430eb91c26161e8d8d11 in term 0.
I20260620 02:57:05.962280 18805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15a6867c87b4feb9dd33fe3f19e63fe" candidate_uuid: "3606b16577d7430eb91c26161e8d8d11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ca62222b98e4ba5b15a97d37ca685c4" is_pre_election: true
I20260620 02:57:05.963037 18805 raft_consensus.cc:2454] T c15a6867c87b4feb9dd33fe3f19e63fe P 2ca62222b98e4ba5b15a97d37ca685c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3606b16577d7430eb91c26161e8d8d11 in term 0.
I20260620 02:57:05.963333 18875 leader_election.cc:302] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3606b16577d7430eb91c26161e8d8d11, 78eb0eb4842642638f7dbe2f7041343f; no voters:
I20260620 02:57:05.963883 19141 raft_consensus.cc:2781] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:57:05.964066 19141 raft_consensus.cc:482] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:57:05.964210 19141 raft_consensus.cc:3037] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:05.966466 19141 raft_consensus.cc:504] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3606b16577d7430eb91c26161e8d8d11" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44375 } } peers { permanent_uuid: "2ca62222b98e4ba5b15a97d37ca685c4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36093 } } peers { permanent_uuid: "78eb0eb4842642638f7dbe2f7041343f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46153 } }
I20260620 02:57:05.967128 19141 leader_election.cc:288] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11 [CANDIDATE]: Term 1 election: Requested vote from peers 2ca62222b98e4ba5b15a97d37ca685c4 (127.0.97.1:36093), 78eb0eb4842642638f7dbe2f7041343f (127.0.97.3:46153)
I20260620 02:57:05.967613 18805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15a6867c87b4feb9dd33fe3f19e63fe" candidate_uuid: "3606b16577d7430eb91c26161e8d8d11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ca62222b98e4ba5b15a97d37ca685c4"
I20260620 02:57:05.967710 19072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15a6867c87b4feb9dd33fe3f19e63fe" candidate_uuid: "3606b16577d7430eb91c26161e8d8d11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78eb0eb4842642638f7dbe2f7041343f"
I20260620 02:57:05.967832 18805 raft_consensus.cc:3037] T c15a6867c87b4feb9dd33fe3f19e63fe P 2ca62222b98e4ba5b15a97d37ca685c4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:05.967957 19072 raft_consensus.cc:3037] T c15a6867c87b4feb9dd33fe3f19e63fe P 78eb0eb4842642638f7dbe2f7041343f [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:05.970156 19072 raft_consensus.cc:2454] T c15a6867c87b4feb9dd33fe3f19e63fe P 78eb0eb4842642638f7dbe2f7041343f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3606b16577d7430eb91c26161e8d8d11 in term 1.
I20260620 02:57:05.970162 18805 raft_consensus.cc:2454] T c15a6867c87b4feb9dd33fe3f19e63fe P 2ca62222b98e4ba5b15a97d37ca685c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3606b16577d7430eb91c26161e8d8d11 in term 1.
I20260620 02:57:05.970621 18872 leader_election.cc:302] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca62222b98e4ba5b15a97d37ca685c4, 3606b16577d7430eb91c26161e8d8d11; no voters:
I20260620 02:57:05.970909 19141 raft_consensus.cc:2781] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:57:05.971356 19141 raft_consensus.cc:686] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11 [term 1 LEADER]: Becoming Leader. State: Replica: 3606b16577d7430eb91c26161e8d8d11, State: Running, Role: LEADER
I20260620 02:57:05.971933 19141 consensus_queue.cc:237] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3606b16577d7430eb91c26161e8d8d11" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44375 } } peers { permanent_uuid: "2ca62222b98e4ba5b15a97d37ca685c4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36093 } } peers { permanent_uuid: "78eb0eb4842642638f7dbe2f7041343f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46153 } }
I20260620 02:57:05.976776 18660 catalog_manager.cc:5495] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3606b16577d7430eb91c26161e8d8d11 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "3606b16577d7430eb91c26161e8d8d11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3606b16577d7430eb91c26161e8d8d11" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca62222b98e4ba5b15a97d37ca685c4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78eb0eb4842642638f7dbe2f7041343f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46153 } health_report { overall_health: UNKNOWN } } }
I20260620 02:57:06.097695 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:06.091884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44622 (local address 127.0.97.2:44375)
0620 02:57:06.092516 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:06.092524 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:06.092553 (+ 29us) server_negotiation.cc:407] Connection header received
0620 02:57:06.092644 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:06.092648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:06.092729 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:06.092891 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:57:06.093347 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:06.094004 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:06.095461 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:06.095859 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:06.095997 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:06.096605 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:57:06.097100 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:57:06.097282 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:06.097394 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 02:57:06.097441 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":356,"threads_started":1}
I20260620 02:57:06.106225 19072 raft_consensus.cc:1261] T c15a6867c87b4feb9dd33fe3f19e63fe P 78eb0eb4842642638f7dbe2f7041343f [term 1 FOLLOWER]: Refusing update from remote peer 3606b16577d7430eb91c26161e8d8d11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:57:06.106259 18805 raft_consensus.cc:1261] T c15a6867c87b4feb9dd33fe3f19e63fe P 2ca62222b98e4ba5b15a97d37ca685c4 [term 1 FOLLOWER]: Refusing update from remote peer 3606b16577d7430eb91c26161e8d8d11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:57:06.107292 19141 consensus_queue.cc:1035] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca62222b98e4ba5b15a97d37ca685c4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:57:06.107606 19146 consensus_queue.cc:1035] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78eb0eb4842642638f7dbe2f7041343f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:57:06.119031 19151 mvcc.cc:205] Tried to move back new op lower bound from 7298761630120570880 to 7298761629585387520. Current Snapshot: MvccSnapshot[applied={T|T < 7298761630120570880}]
I20260620 02:57:06.120740 19154 mvcc.cc:205] Tried to move back new op lower bound from 7298761630120570880 to 7298761629585387520. Current Snapshot: MvccSnapshot[applied={T|T < 7298761630120570880}]
I20260620 02:57:06.141185 18661 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:57:06.141510 18661 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:57:06.145558 18661 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de9f09fcf5cd4bb6a168e217a6d7bdb9: Sending DeleteTablet for 3 replicas of tablet c15a6867c87b4feb9dd33fe3f19e63fe
I20260620 02:57:06.146986 18785 tablet_service.cc:1514] Processing DeleteTablet for tablet c15a6867c87b4feb9dd33fe3f19e63fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:57:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53926
I20260620 02:57:06.147280 18919 tablet_service.cc:1514] Processing DeleteTablet for tablet c15a6867c87b4feb9dd33fe3f19e63fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:57:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44610
I20260620 02:57:06.147364 19052 tablet_service.cc:1514] Processing DeleteTablet for tablet c15a6867c87b4feb9dd33fe3f19e63fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:57:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44834
I20260620 02:57:06.149010 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 18722
I20260620 02:57:06.158380 19163 tablet_replica.cc:330] T c15a6867c87b4feb9dd33fe3f19e63fe P 2ca62222b98e4ba5b15a97d37ca685c4: stopping tablet replica
I20260620 02:57:06.159116 19163 raft_consensus.cc:2229] T c15a6867c87b4feb9dd33fe3f19e63fe P 2ca62222b98e4ba5b15a97d37ca685c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:57:06.159857 19163 raft_consensus.cc:2258] T c15a6867c87b4feb9dd33fe3f19e63fe P 2ca62222b98e4ba5b15a97d37ca685c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:57:06.162653 19163 ts_tablet_manager.cc:1905] T c15a6867c87b4feb9dd33fe3f19e63fe P 2ca62222b98e4ba5b15a97d37ca685c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:57:06.163262 19164 tablet_replica.cc:330] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11: stopping tablet replica
I20260620 02:57:06.163853 19164 raft_consensus.cc:2229] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:57:06.164737 19164 raft_consensus.cc:2258] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:57:06.166836 19164 ts_tablet_manager.cc:1905] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:57:06.174772 19164 ts_tablet_manager.cc:1918] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:57:06.175066 19164 log.cc:1196] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-1/wal/wals/c15a6867c87b4feb9dd33fe3f19e63fe
I20260620 02:57:06.175735 19164 ts_tablet_manager.cc:1939] T c15a6867c87b4feb9dd33fe3f19e63fe P 3606b16577d7430eb91c26161e8d8d11: Deleting consensus metadata
I20260620 02:57:06.177819 18648 catalog_manager.cc:4841] TS 3606b16577d7430eb91c26161e8d8d11 (127.0.97.2:44375): tablet c15a6867c87b4feb9dd33fe3f19e63fe (table test-table [id=dff0ac55569c4f398a73bcab37e08980]) successfully deleted
I20260620 02:57:06.178869 19165 tablet_replica.cc:330] T c15a6867c87b4feb9dd33fe3f19e63fe P 78eb0eb4842642638f7dbe2f7041343f: stopping tablet replica
I20260620 02:57:06.179366 19165 raft_consensus.cc:2229] T c15a6867c87b4feb9dd33fe3f19e63fe P 78eb0eb4842642638f7dbe2f7041343f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:57:06.179908 19165 raft_consensus.cc:2258] T c15a6867c87b4feb9dd33fe3f19e63fe P 78eb0eb4842642638f7dbe2f7041343f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:57:06.181931 19165 ts_tablet_manager.cc:1905] T c15a6867c87b4feb9dd33fe3f19e63fe P 78eb0eb4842642638f7dbe2f7041343f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:57:06.181937 19163 ts_tablet_manager.cc:1918] T c15a6867c87b4feb9dd33fe3f19e63fe P 2ca62222b98e4ba5b15a97d37ca685c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:57:06.182155 19163 log.cc:1196] T c15a6867c87b4feb9dd33fe3f19e63fe P 2ca62222b98e4ba5b15a97d37ca685c4: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-0/wal/wals/c15a6867c87b4feb9dd33fe3f19e63fe
I20260620 02:57:06.182734 19163 ts_tablet_manager.cc:1939] T c15a6867c87b4feb9dd33fe3f19e63fe P 2ca62222b98e4ba5b15a97d37ca685c4: Deleting consensus metadata
I20260620 02:57:06.184443 18645 catalog_manager.cc:4841] TS 2ca62222b98e4ba5b15a97d37ca685c4 (127.0.97.1:36093): tablet c15a6867c87b4feb9dd33fe3f19e63fe (table test-table [id=dff0ac55569c4f398a73bcab37e08980]) successfully deleted
Jun 20 02:57:06 dist-test-slave-8p8c krb5kdc[18616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:57:06.189226 19165 ts_tablet_manager.cc:1918] T c15a6867c87b4feb9dd33fe3f19e63fe P 78eb0eb4842642638f7dbe2f7041343f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:57:06.189446 19165 log.cc:1196] T c15a6867c87b4feb9dd33fe3f19e63fe P 78eb0eb4842642638f7dbe2f7041343f: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781923914418765-388-0/minicluster-data/ts-2/wal/wals/c15a6867c87b4feb9dd33fe3f19e63fe
I20260620 02:57:06.190014 19165 ts_tablet_manager.cc:1939] T c15a6867c87b4feb9dd33fe3f19e63fe P 78eb0eb4842642638f7dbe2f7041343f: Deleting consensus metadata
I20260620 02:57:06.191972 18648 catalog_manager.cc:4841] TS 78eb0eb4842642638f7dbe2f7041343f (127.0.97.3:46153): tablet c15a6867c87b4feb9dd33fe3f19e63fe (table test-table [id=dff0ac55569c4f398a73bcab37e08980]) successfully deleted
I20260620 02:57:06.194509 19132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:06.150815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53932 (local address 127.0.97.1:36093)
0620 02:57:06.151051 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:06.151069 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:06.160837 (+ 9768us) server_negotiation.cc:407] Connection header received
0620 02:57:06.170835 (+ 9998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:06.170845 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:06.170997 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:06.171089 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:06.173944 (+ 2855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:06.175469 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:06.178170 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:06.178838 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:06.178916 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:06.188054 (+ 9138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:06.188088 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:06.188104 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:06.188145 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:06.191407 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:06.192300 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:06.192308 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:06.192310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:06.192441 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:06.192850 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:06.192855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:06.192858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:06.193235 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:06.193546 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:06.193870 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 02:57:06.194095 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 02:57:06.195189 18845 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:57:06.286568 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 18722
I20260620 02:57:06.308856 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 18856
Jun 20 02:57:06 dist-test-slave-8p8c krb5kdc[18616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:57:06.326836 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:06.309838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44634 (local address 127.0.97.2:44375)
0620 02:57:06.310065 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:06.310071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:06.310092 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:57:06.310159 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:06.310165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:06.310245 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:06.310353 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:06.311727 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:06.312947 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:06.314213 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:06.314942 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:06.315083 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:06.319660 (+ 4577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:06.319682 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:06.319703 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:06.319759 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:06.323876 (+ 4117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:06.324603 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:06.324613 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:06.324617 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:06.324795 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:06.325143 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:06.325150 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:06.325153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:06.325619 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:06.325921 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:06.326134 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 02:57:06.326417 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 02:57:06.327783 18979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:57:06.456796 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 18856
I20260620 02:57:06.476593 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 18989
Jun 20 02:57:06 dist-test-slave-8p8c krb5kdc[18616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:57:06.494104 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:06.477533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44844 (local address 127.0.97.3:46153)
0620 02:57:06.478024 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:06.478028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:06.478045 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:57:06.478101 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:06.478104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:06.478183 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:06.478282 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:06.479650 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:06.480425 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:06.482103 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:06.482704 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:06.482779 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:06.487560 (+ 4781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:06.487593 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:06.487609 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:06.487656 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:06.490972 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:06.491743 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:06.491752 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:06.491754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:06.491892 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:06.492350 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:06.492355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:06.492357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:06.492750 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:06.492991 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:06.493350 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 02:57:06.493816 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":285,"threads_started":1}
I20260620 02:57:06.494843 19112 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:57:06.585914 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 18989
I20260620 02:57:06.602721 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 18629
I20260620 02:57:06.603978 18691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:57:06.717465 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 18629
2026-06-20T02:57:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4788 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T02:57:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:57:06Z Disabled control of system clock
I20260620 02:57:06.798424 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34331
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 02:57:07.145043 19175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:07.145352 19175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:07.145522 19175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:07.153100 19175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:57:07.153204 19175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:07.153241 19175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:07.153290 19175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:57:07.153334 19175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:57:07.163053 19175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34331
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:07.164350 19175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:07.166149 19175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:07.177207 19180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:07.178680 19183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:07.178797 19181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:07.179559 19175 server_base.cc:1034] running on GCE node
I20260620 02:57:07.180444 19175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:07.181891 19175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:07.183169 19175 hybrid_clock.cc:648] HybridClock initialized: now 1781924227183115 us; error 47 us; skew 500 ppm
I20260620 02:57:07.183602 19175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:57:07.186188 19175 webserver.cc:460] Webserver started at http://127.0.97.62:45399/ using document root <none> and password file <none>
I20260620 02:57:07.186995 19175 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:07.187119 19175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:07.187479 19175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:07.190599 19175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "56e0e14ecc5b49b2b21e380e39d114ad"
format_stamp: "Formatted at 2026-06-20 02:57:07 on dist-test-slave-8p8c"
server_key: "eda6555306140c8feac72d0551a50166"
server_key_iv: "790f5d1c9f9966b5a006dd85b84d7025"
server_key_version: "encryptionkey@0"
I20260620 02:57:07.191438 19175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "56e0e14ecc5b49b2b21e380e39d114ad"
format_stamp: "Formatted at 2026-06-20 02:57:07 on dist-test-slave-8p8c"
server_key: "eda6555306140c8feac72d0551a50166"
server_key_iv: "790f5d1c9f9966b5a006dd85b84d7025"
server_key_version: "encryptionkey@0"
I20260620 02:57:07.197029 19175 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 02:57:07.200812 19189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:07.202358 19175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 02:57:07.202625 19175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "56e0e14ecc5b49b2b21e380e39d114ad"
format_stamp: "Formatted at 2026-06-20 02:57:07 on dist-test-slave-8p8c"
server_key: "eda6555306140c8feac72d0551a50166"
server_key_iv: "790f5d1c9f9966b5a006dd85b84d7025"
server_key_version: "encryptionkey@0"
I20260620 02:57:07.202850 19175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:07.221632 19175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:07.222590 19175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:07.222916 19175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:07.242622 19175 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34331
I20260620 02:57:07.242650 19240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34331 every 8 connection(s)
I20260620 02:57:07.244136 19175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:57:07.248206 19241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:07.253559 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 19175
I20260620 02:57:07.253762 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:57:07.254218 388 external_mini_cluster.cc:1428] Setting key c78c7f792c3e26a5c0ed072f7b8f2b4c
I20260620 02:57:07.259728 19241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56e0e14ecc5b49b2b21e380e39d114ad: Bootstrap starting.
I20260620 02:57:07.263655 19241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56e0e14ecc5b49b2b21e380e39d114ad: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:07.264889 19241 log.cc:824] T 00000000000000000000000000000000 P 56e0e14ecc5b49b2b21e380e39d114ad: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:07.268404 19241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56e0e14ecc5b49b2b21e380e39d114ad: No bootstrap required, opened a new log
I20260620 02:57:07.269537 19244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:07.256335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59974 (local address 127.0.97.62:34331)
0620 02:57:07.257488 (+ 1153us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:07.257503 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:07.257552 (+ 49us) server_negotiation.cc:407] Connection header received
0620 02:57:07.259318 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:07.259341 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:07.259896 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:07.260243 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:07.260899 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:07.262464 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:07.264539 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:07.265319 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:07.265559 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:07.266737 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:07.266747 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:07.266766 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:07.267467 (+ 701us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:07.267579 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:07.267604 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:07.267721 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:07.267932 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 02:57:07.268272 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":428,"threads_started":1}
I20260620 02:57:07.276154 19241 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56e0e14ecc5b49b2b21e380e39d114ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e0e14ecc5b49b2b21e380e39d114ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34331 } }
I20260620 02:57:07.276491 19241 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56e0e14ecc5b49b2b21e380e39d114ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:07.276554 19241 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56e0e14ecc5b49b2b21e380e39d114ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e0e14ecc5b49b2b21e380e39d114ad, State: Initialized, Role: FOLLOWER
I20260620 02:57:07.277154 19241 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56e0e14ecc5b49b2b21e380e39d114ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e0e14ecc5b49b2b21e380e39d114ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34331 } }
I20260620 02:57:07.277367 19241 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56e0e14ecc5b49b2b21e380e39d114ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:57:07.277463 19241 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56e0e14ecc5b49b2b21e380e39d114ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:57:07.277586 19241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56e0e14ecc5b49b2b21e380e39d114ad [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:07.280139 19241 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56e0e14ecc5b49b2b21e380e39d114ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e0e14ecc5b49b2b21e380e39d114ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34331 } }
I20260620 02:57:07.280624 19241 leader_election.cc:302] T 00000000000000000000000000000000 P 56e0e14ecc5b49b2b21e380e39d114ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56e0e14ecc5b49b2b21e380e39d114ad; no voters:
I20260620 02:57:07.281257 19241 leader_election.cc:288] T 00000000000000000000000000000000 P 56e0e14ecc5b49b2b21e380e39d114ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:57:07.281415 19246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56e0e14ecc5b49b2b21e380e39d114ad [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:57:07.282198 19246 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56e0e14ecc5b49b2b21e380e39d114ad [term 1 LEADER]: Becoming Leader. State: Replica: 56e0e14ecc5b49b2b21e380e39d114ad, State: Running, Role: LEADER
I20260620 02:57:07.283003 19246 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56e0e14ecc5b49b2b21e380e39d114ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e0e14ecc5b49b2b21e380e39d114ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34331 } }
I20260620 02:57:07.283567 19241 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56e0e14ecc5b49b2b21e380e39d114ad [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:57:07.285655 19247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56e0e14ecc5b49b2b21e380e39d114ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56e0e14ecc5b49b2b21e380e39d114ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e0e14ecc5b49b2b21e380e39d114ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34331 } } }
I20260620 02:57:07.286105 19247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56e0e14ecc5b49b2b21e380e39d114ad [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:07.287182 19248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56e0e14ecc5b49b2b21e380e39d114ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56e0e14ecc5b49b2b21e380e39d114ad. Latest consensus state: current_term: 1 leader_uuid: "56e0e14ecc5b49b2b21e380e39d114ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e0e14ecc5b49b2b21e380e39d114ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34331 } } }
I20260620 02:57:07.287817 19248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56e0e14ecc5b49b2b21e380e39d114ad [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:07.290872 19252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:57:07.296655 19252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:57:07.306237 19252 catalog_manager.cc:1347] Generated new cluster ID: 2c6b86201e704472af3b852639014566
I20260620 02:57:07.306384 19252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:57:07.328122 19252 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:57:07.330026 19252 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:57:07.341821 19252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56e0e14ecc5b49b2b21e380e39d114ad: Generated new TSK 0
I20260620 02:57:07.342785 19252 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:57:07.358593 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34331
--builtin_ntp_servers=127.0.97.20:40287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 02:57:07.718544 19265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:07.718815 19265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:07.718961 19265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:07.726298 19265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:07.726466 19265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:07.726636 19265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:57:07.735739 19265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:07.737061 19265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:07.738847 19265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:07.752300 19273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:07.752321 19271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:07.752265 19270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:07.753121 19265 server_base.cc:1034] running on GCE node
I20260620 02:57:07.753686 19265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:07.754693 19265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:07.755965 19265 hybrid_clock.cc:648] HybridClock initialized: now 1781924227755889 us; error 52 us; skew 500 ppm
I20260620 02:57:07.756382 19265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:57:07.758788 19265 webserver.cc:460] Webserver started at http://127.0.97.1:33393/ using document root <none> and password file <none>
I20260620 02:57:07.759498 19265 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:07.759622 19265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:07.759963 19265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:07.763108 19265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-0/data/instance:
uuid: "60d4730368534761b1cf32cbb0a9e06f"
format_stamp: "Formatted at 2026-06-20 02:57:07 on dist-test-slave-8p8c"
server_key: "0b557b3d02f23bbd7d65ceaf9236b5c1"
server_key_iv: "fff45c72fb70a166f82824bd1edddccb"
server_key_version: "encryptionkey@0"
I20260620 02:57:07.763962 19265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-0/wal/instance:
uuid: "60d4730368534761b1cf32cbb0a9e06f"
format_stamp: "Formatted at 2026-06-20 02:57:07 on dist-test-slave-8p8c"
server_key: "0b557b3d02f23bbd7d65ceaf9236b5c1"
server_key_iv: "fff45c72fb70a166f82824bd1edddccb"
server_key_version: "encryptionkey@0"
I20260620 02:57:07.769860 19265 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260620 02:57:07.773643 19279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:07.775146 19265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:57:07.775316 19265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-0/wal
uuid: "60d4730368534761b1cf32cbb0a9e06f"
format_stamp: "Formatted at 2026-06-20 02:57:07 on dist-test-slave-8p8c"
server_key: "0b557b3d02f23bbd7d65ceaf9236b5c1"
server_key_iv: "fff45c72fb70a166f82824bd1edddccb"
server_key_version: "encryptionkey@0"
I20260620 02:57:07.775538 19265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:07.797676 19265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:07.798564 19265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:07.798839 19265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:07.799793 19265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:57:07.801259 19265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:57:07.801383 19265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:07.801538 19265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:57:07.801638 19265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:07.835348 19265 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43821
I20260620 02:57:07.835436 19391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43821 every 8 connection(s)
I20260620 02:57:07.836674 19265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:57:07.837990 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 19265
I20260620 02:57:07.838165 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:57:07.838775 388 external_mini_cluster.cc:1428] Setting key 217f511728d81197574fe485b81c9feb
I20260620 02:57:07.841908 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34331
--builtin_ntp_servers=127.0.97.20:40287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 02:57:07.853204 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:07.839545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59177 (local address 127.0.97.62:34331)
0620 02:57:07.840063 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:07.840070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:07.841424 (+ 1354us) server_negotiation.cc:407] Connection header received
0620 02:57:07.844425 (+ 3001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:07.844429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:07.844499 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:07.844642 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:07.846160 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:07.847020 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:07.849602 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:07.850082 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:07.850191 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:07.852006 (+ 1815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:07.852015 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:07.852024 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:07.852053 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:07.852113 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:07.852137 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:07.852259 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:07.852563 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 02:57:07.852699 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":293,"threads_started":1}
I20260620 02:57:07.853969 19395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:07.840197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34331 (local address 127.0.97.1:59177)
0620 02:57:07.841285 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:07.841337 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:07.844079 (+ 2742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:07.844960 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:07.844975 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:07.845107 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:57:07.845939 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:07.845966 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:07.847191 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:07.847200 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:07.849417 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:07.849437 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:07.850332 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:07.850342 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:07.850742 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:07.851630 (+ 888us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:57:07.851632 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:57:07.851676 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:57:07.851763 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:57:07.852304 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:57:07.852313 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:57:07.852320 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 02:57:07.852517 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 02:57:07.852857 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":314,"threads_started":1}
I20260620 02:57:07.855829 19392 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34331
I20260620 02:57:07.856170 19392 heartbeater.cc:464] Registering TS with master...
I20260620 02:57:07.856952 19392 heartbeater.cc:511] Master 127.0.97.62:34331 requested a full tablet report, sending...
I20260620 02:57:07.859464 19206 ts_manager.cc:194] Registered new tserver with Master: 60d4730368534761b1cf32cbb0a9e06f (127.0.97.1:43821)
I20260620 02:57:07.861860 19206 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:59177
W20260620 02:57:08.193822 19396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:08.194146 19396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:08.194299 19396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:08.201538 19396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:08.201633 19396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:08.201804 19396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:57:08.210924 19396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:08.212227 19396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:08.213990 19396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:08.227298 19401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:08.227299 19402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:08.227651 19404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:08.228700 19396 server_base.cc:1034] running on GCE node
I20260620 02:57:08.229358 19396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:08.230305 19396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:08.231590 19396 hybrid_clock.cc:648] HybridClock initialized: now 1781924228231528 us; error 43 us; skew 500 ppm
I20260620 02:57:08.231967 19396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:57:08.234422 19396 webserver.cc:460] Webserver started at http://127.0.97.2:38389/ using document root <none> and password file <none>
I20260620 02:57:08.235176 19396 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:08.235322 19396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:08.235667 19396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:08.238723 19396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-1/data/instance:
uuid: "abe71a1ade2c4d189bd7161772a48fc6"
format_stamp: "Formatted at 2026-06-20 02:57:08 on dist-test-slave-8p8c"
server_key: "9198cf59d7bc85f37b58f7ca09489104"
server_key_iv: "cabcc4bb2d75881dbd3547ffaaf48d97"
server_key_version: "encryptionkey@0"
I20260620 02:57:08.239579 19396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-1/wal/instance:
uuid: "abe71a1ade2c4d189bd7161772a48fc6"
format_stamp: "Formatted at 2026-06-20 02:57:08 on dist-test-slave-8p8c"
server_key: "9198cf59d7bc85f37b58f7ca09489104"
server_key_iv: "cabcc4bb2d75881dbd3547ffaaf48d97"
server_key_version: "encryptionkey@0"
I20260620 02:57:08.245898 19396 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 02:57:08.250567 19410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:08.252148 19396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:57:08.252314 19396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-1/wal
uuid: "abe71a1ade2c4d189bd7161772a48fc6"
format_stamp: "Formatted at 2026-06-20 02:57:08 on dist-test-slave-8p8c"
server_key: "9198cf59d7bc85f37b58f7ca09489104"
server_key_iv: "cabcc4bb2d75881dbd3547ffaaf48d97"
server_key_version: "encryptionkey@0"
I20260620 02:57:08.252560 19396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:08.272574 19396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:08.273428 19396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:08.273752 19396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:08.274801 19396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:57:08.276425 19396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:57:08.276551 19396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:08.276670 19396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:57:08.276785 19396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:08.313982 19396 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37267
I20260620 02:57:08.314028 19522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37267 every 8 connection(s)
I20260620 02:57:08.315490 19396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:57:08.322913 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 19396
I20260620 02:57:08.323153 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:57:08.323684 388 external_mini_cluster.cc:1428] Setting key bbb2e573fd96afd95172dde02362bb2e
I20260620 02:57:08.327083 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34331
--builtin_ntp_servers=127.0.97.20:40287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 02:57:08.329282 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:08.318240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41139 (local address 127.0.97.62:34331)
0620 02:57:08.318481 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:08.318487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:08.319900 (+ 1413us) server_negotiation.cc:407] Connection header received
0620 02:57:08.321664 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:08.321669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:08.321740 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:08.321834 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:08.322989 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:08.324094 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:08.325903 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:08.326599 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:08.326787 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:08.328226 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:08.328237 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:08.328245 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:08.328273 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:08.328322 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:08.328343 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:08.328463 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:08.328729 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 02:57:08.328848 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 02:57:08.330178 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:08.318854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34331 (local address 127.0.97.2:41139)
0620 02:57:08.319740 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:08.319776 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:08.321378 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:08.322033 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:08.322040 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:08.322155 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:57:08.322794 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:08.322811 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:08.324259 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:08.324266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:08.325706 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:08.325720 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:08.326831 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:08.326836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:08.327041 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:08.327870 (+ 829us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:57:08.327873 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:57:08.327913 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:57:08.327997 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:57:08.328486 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:57:08.328495 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:57:08.328502 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 02:57:08.328673 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 02:57:08.329015 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":287,"threads_started":1}
I20260620 02:57:08.331782 19523 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34331
I20260620 02:57:08.332122 19523 heartbeater.cc:464] Registering TS with master...
I20260620 02:57:08.332871 19523 heartbeater.cc:511] Master 127.0.97.62:34331 requested a full tablet report, sending...
I20260620 02:57:08.334362 19206 ts_manager.cc:194] Registered new tserver with Master: abe71a1ade2c4d189bd7161772a48fc6 (127.0.97.2:37267)
I20260620 02:57:08.335208 19206 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:41139
W20260620 02:57:08.660854 19526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:08.661134 19526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:08.661223 19526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:08.668653 19526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:08.668803 19526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:08.668988 19526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:57:08.677899 19526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:08.679335 19526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:08.681226 19526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:08.694577 19531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:08.694670 19532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:08.694690 19534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:08.695724 19526 server_base.cc:1034] running on GCE node
I20260620 02:57:08.696344 19526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:08.697331 19526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:08.698542 19526 hybrid_clock.cc:648] HybridClock initialized: now 1781924228698489 us; error 49 us; skew 500 ppm
I20260620 02:57:08.698863 19526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:57:08.705566 19526 webserver.cc:460] Webserver started at http://127.0.97.3:35937/ using document root <none> and password file <none>
I20260620 02:57:08.706496 19526 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:08.706650 19526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:08.707019 19526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:08.710186 19526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-2/data/instance:
uuid: "e0f142cb6cee44de9b506b786637fbe2"
format_stamp: "Formatted at 2026-06-20 02:57:08 on dist-test-slave-8p8c"
server_key: "0135170108b630911f7bc274e677bcb4"
server_key_iv: "1176b842d898f5c80dbceb2de5caca7d"
server_key_version: "encryptionkey@0"
I20260620 02:57:08.711064 19526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e0f142cb6cee44de9b506b786637fbe2"
format_stamp: "Formatted at 2026-06-20 02:57:08 on dist-test-slave-8p8c"
server_key: "0135170108b630911f7bc274e677bcb4"
server_key_iv: "1176b842d898f5c80dbceb2de5caca7d"
server_key_version: "encryptionkey@0"
I20260620 02:57:08.716756 19526 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:57:08.720530 19540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:08.721987 19526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:57:08.722208 19526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-2/wal
uuid: "e0f142cb6cee44de9b506b786637fbe2"
format_stamp: "Formatted at 2026-06-20 02:57:08 on dist-test-slave-8p8c"
server_key: "0135170108b630911f7bc274e677bcb4"
server_key_iv: "1176b842d898f5c80dbceb2de5caca7d"
server_key_version: "encryptionkey@0"
I20260620 02:57:08.722471 19526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:08.745080 19526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:08.746001 19526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:08.746308 19526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:08.747394 19526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:57:08.748921 19526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:57:08.749043 19526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:08.749177 19526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:57:08.749264 19526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:08.785084 19526 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39473
I20260620 02:57:08.785264 19652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39473 every 8 connection(s)
I20260620 02:57:08.786405 19526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:57:08.794126 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 19526
I20260620 02:57:08.794323 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:57:08.794914 388 external_mini_cluster.cc:1428] Setting key 2b1f3d2b229c1abb3551e85ecc5d969e
I20260620 02:57:08.801028 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:08.789117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54797 (local address 127.0.97.62:34331)
0620 02:57:08.789305 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:08.789309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:08.790785 (+ 1476us) server_negotiation.cc:407] Connection header received
0620 02:57:08.792813 (+ 2028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:08.792820 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:08.792913 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:08.793072 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:08.794647 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:08.796074 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:08.798059 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:08.798589 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:08.798693 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:08.800201 (+ 1508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:08.800206 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:08.800210 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:08.800228 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:08.800272 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:08.800292 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:08.800382 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:08.800687 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 02:57:08.800772 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 02:57:08.802126 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:08.789591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34331 (local address 127.0.97.3:54797)
0620 02:57:08.790625 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:08.790669 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:08.792492 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:08.793361 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:08.793378 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:08.793526 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:57:08.794429 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:08.794458 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:08.796301 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:08.796307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:08.797905 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:08.797917 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:08.798804 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:08.798809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:08.799069 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:08.799838 (+ 769us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:57:08.799841 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:57:08.799882 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:57:08.799977 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:57:08.800441 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:57:08.800451 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:57:08.800458 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 02:57:08.800634 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 02:57:08.800976 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":315,"threads_started":1}
I20260620 02:57:08.803692 19653 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34331
I20260620 02:57:08.804013 19653 heartbeater.cc:464] Registering TS with master...
I20260620 02:57:08.804836 19653 heartbeater.cc:511] Master 127.0.97.62:34331 requested a full tablet report, sending...
I20260620 02:57:08.806444 19206 ts_manager.cc:194] Registered new tserver with Master: e0f142cb6cee44de9b506b786637fbe2 (127.0.97.3:39473)
I20260620 02:57:08.807355 19206 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:54797
I20260620 02:57:08.810706 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:57:08.821421 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:08.814969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59982 (local address 127.0.97.62:34331)
0620 02:57:08.815202 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:08.815210 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:08.815617 (+ 407us) server_negotiation.cc:407] Connection header received
0620 02:57:08.815817 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:08.815823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:08.815905 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:08.816013 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:08.816459 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:08.817165 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:08.818608 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:08.819016 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:08.819157 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:08.820442 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:08.820448 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:08.820452 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:08.820472 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:08.820511 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:08.820526 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:08.820699 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:08.820971 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 02:57:08.821038 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 02:57:08.826252 19206 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:57:08.833693 19206 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:57:08.861358 19667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:08.849186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39473 (local address 127.0.0.1:39490)
0620 02:57:08.849737 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:08.849755 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:08.849861 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:08.851210 (+ 1349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:08.851217 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:08.851253 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:57:08.851801 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:08.851816 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:08.854740 (+ 2924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:08.854753 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:08.857355 (+ 2602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:08.857368 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:08.859717 (+ 2349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:08.859725 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:08.860644 (+ 919us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:08.860678 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 02:57:08.860773 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 02:57:08.860958 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":4,"thread_start_us":278,"threads_started":1}
I20260620 02:57:08.861377 19666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:08.848229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38456 (local address 127.0.97.1:43821)
0620 02:57:08.848870 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:08.848880 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:08.849038 (+ 158us) server_negotiation.cc:407] Connection header received
0620 02:57:08.849477 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:08.849484 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:08.849768 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:08.849995 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:57:08.850855 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:08.852925 (+ 2070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:08.858361 (+ 5436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:08.859818 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:08.859963 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:08.860253 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:08.860660 (+ 407us) server_negotiation.cc:299] Negotiation successful
0620 02:57:08.860843 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":329,"threads_started":1}
I20260620 02:57:08.862142 19665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:08.847990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43821 (local address 127.0.0.1:38456)
0620 02:57:08.848875 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:08.848917 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:08.849179 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:08.850307 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:08.850317 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:08.850346 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:57:08.850687 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:08.850695 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:08.853141 (+ 2446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:08.853149 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:08.858132 (+ 4983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:08.858155 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:08.860087 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:08.860093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:08.860357 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:08.860400 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 02:57:08.861609 (+ 1209us) client_negotiation.cc:240] Negotiation successful
0620 02:57:08.861761 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":2070,"thread_start_us":305,"threads_started":1}
I20260620 02:57:08.862195 19668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:08.849542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39490 (local address 127.0.97.3:39473)
0620 02:57:08.850487 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:08.850499 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:08.850555 (+ 56us) server_negotiation.cc:407] Connection header received
0620 02:57:08.850670 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:08.850681 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:08.850957 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:08.852029 (+ 1072us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:57:08.852388 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:08.854543 (+ 2155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:08.857907 (+ 3364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:08.859474 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:08.859635 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:08.859924 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:08.861620 (+ 1696us) server_negotiation.cc:299] Negotiation successful
0620 02:57:08.861798 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":550,"threads_started":1}
I20260620 02:57:08.865559 19392 heartbeater.cc:503] Master 127.0.97.62:34331 was elected leader, sending a full tablet report...
I20260620 02:57:08.866291 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:08.850973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37267 (local address 127.0.0.1:33762)
0620 02:57:08.856064 (+ 5091us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:08.856082 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:08.856197 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:08.857509 (+ 1312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:08.857514 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:08.857552 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:57:08.858044 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:08.858061 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:08.859482 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:08.859490 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:08.861990 (+ 2500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:08.862008 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:08.865457 (+ 3449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:08.865463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:08.865738 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:08.865769 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 02:57:08.865857 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 02:57:08.865985 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4962,"mutex_wait_us":57,"thread_start_us":4822,"threads_started":1}
I20260620 02:57:08.867115 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:08.850078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33762 (local address 127.0.97.2:37267)
0620 02:57:08.857041 (+ 6963us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:08.857048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:08.857073 (+ 25us) server_negotiation.cc:407] Connection header received
0620 02:57:08.857155 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:08.857160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:08.857331 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:08.857532 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:57:08.858244 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:08.859345 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:08.862206 (+ 2861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:08.864953 (+ 2747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:08.865084 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:08.865328 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:08.866615 (+ 1287us) server_negotiation.cc:299] Negotiation successful
0620 02:57:08.866804 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6793,"thread_start_us":6690,"threads_started":1}
I20260620 02:57:08.866925 19327 tablet_service.cc:1467] Processing CreateTablet for tablet 85c101ce9285464686b12c71e3c3d2e5 (DEFAULT_TABLE table=test-table [id=d7b5302ef192408f8096fb9ba50c2be6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:57:08.866793 19588 tablet_service.cc:1467] Processing CreateTablet for tablet 85c101ce9285464686b12c71e3c3d2e5 (DEFAULT_TABLE table=test-table [id=d7b5302ef192408f8096fb9ba50c2be6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:57:08.868263 19327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85c101ce9285464686b12c71e3c3d2e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:08.868433 19588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85c101ce9285464686b12c71e3c3d2e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:08.870836 19458 tablet_service.cc:1467] Processing CreateTablet for tablet 85c101ce9285464686b12c71e3c3d2e5 (DEFAULT_TABLE table=test-table [id=d7b5302ef192408f8096fb9ba50c2be6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:57:08.872162 19458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85c101ce9285464686b12c71e3c3d2e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:08.879573 19671 tablet_bootstrap.cc:492] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2: Bootstrap starting.
I20260620 02:57:08.883147 19671 tablet_bootstrap.cc:654] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:08.883426 19672 tablet_bootstrap.cc:492] T 85c101ce9285464686b12c71e3c3d2e5 P 60d4730368534761b1cf32cbb0a9e06f: Bootstrap starting.
I20260620 02:57:08.884577 19671 log.cc:824] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:08.886725 19674 tablet_bootstrap.cc:492] T 85c101ce9285464686b12c71e3c3d2e5 P abe71a1ade2c4d189bd7161772a48fc6: Bootstrap starting.
I20260620 02:57:08.886907 19672 tablet_bootstrap.cc:654] T 85c101ce9285464686b12c71e3c3d2e5 P 60d4730368534761b1cf32cbb0a9e06f: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:08.888144 19672 log.cc:824] T 85c101ce9285464686b12c71e3c3d2e5 P 60d4730368534761b1cf32cbb0a9e06f: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:08.888547 19671 tablet_bootstrap.cc:492] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2: No bootstrap required, opened a new log
I20260620 02:57:08.888863 19671 ts_tablet_manager.cc:1397] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260620 02:57:08.889753 19674 tablet_bootstrap.cc:654] T 85c101ce9285464686b12c71e3c3d2e5 P abe71a1ade2c4d189bd7161772a48fc6: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:08.891027 19674 log.cc:824] T 85c101ce9285464686b12c71e3c3d2e5 P abe71a1ade2c4d189bd7161772a48fc6: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:08.891633 19672 tablet_bootstrap.cc:492] T 85c101ce9285464686b12c71e3c3d2e5 P 60d4730368534761b1cf32cbb0a9e06f: No bootstrap required, opened a new log
I20260620 02:57:08.891928 19672 ts_tablet_manager.cc:1397] T 85c101ce9285464686b12c71e3c3d2e5 P 60d4730368534761b1cf32cbb0a9e06f: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 02:57:08.894073 19674 tablet_bootstrap.cc:492] T 85c101ce9285464686b12c71e3c3d2e5 P abe71a1ade2c4d189bd7161772a48fc6: No bootstrap required, opened a new log
I20260620 02:57:08.894328 19674 ts_tablet_manager.cc:1397] T 85c101ce9285464686b12c71e3c3d2e5 P abe71a1ade2c4d189bd7161772a48fc6: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 02:57:08.897007 19671 raft_consensus.cc:348] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d4730368534761b1cf32cbb0a9e06f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43821 } } peers { permanent_uuid: "e0f142cb6cee44de9b506b786637fbe2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39473 } } peers { permanent_uuid: "abe71a1ade2c4d189bd7161772a48fc6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37267 } }
I20260620 02:57:08.897369 19671 raft_consensus.cc:374] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:08.897490 19671 raft_consensus.cc:729] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0f142cb6cee44de9b506b786637fbe2, State: Initialized, Role: FOLLOWER
I20260620 02:57:08.898232 19671 consensus_queue.cc:260] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d4730368534761b1cf32cbb0a9e06f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43821 } } peers { permanent_uuid: "e0f142cb6cee44de9b506b786637fbe2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39473 } } peers { permanent_uuid: "abe71a1ade2c4d189bd7161772a48fc6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37267 } }
I20260620 02:57:08.899585 19653 heartbeater.cc:503] Master 127.0.97.62:34331 was elected leader, sending a full tablet report...
I20260620 02:57:08.901402 19671 ts_tablet_manager.cc:1428] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260620 02:57:08.902282 19672 raft_consensus.cc:348] T 85c101ce9285464686b12c71e3c3d2e5 P 60d4730368534761b1cf32cbb0a9e06f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d4730368534761b1cf32cbb0a9e06f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43821 } } peers { permanent_uuid: "e0f142cb6cee44de9b506b786637fbe2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39473 } } peers { permanent_uuid: "abe71a1ade2c4d189bd7161772a48fc6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37267 } }
I20260620 02:57:08.902743 19672 raft_consensus.cc:374] T 85c101ce9285464686b12c71e3c3d2e5 P 60d4730368534761b1cf32cbb0a9e06f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:08.902844 19672 raft_consensus.cc:729] T 85c101ce9285464686b12c71e3c3d2e5 P 60d4730368534761b1cf32cbb0a9e06f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d4730368534761b1cf32cbb0a9e06f, State: Initialized, Role: FOLLOWER
I20260620 02:57:08.903518 19672 consensus_queue.cc:260] T 85c101ce9285464686b12c71e3c3d2e5 P 60d4730368534761b1cf32cbb0a9e06f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d4730368534761b1cf32cbb0a9e06f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43821 } } peers { permanent_uuid: "e0f142cb6cee44de9b506b786637fbe2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39473 } } peers { permanent_uuid: "abe71a1ade2c4d189bd7161772a48fc6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37267 } }
I20260620 02:57:08.904688 19674 raft_consensus.cc:348] T 85c101ce9285464686b12c71e3c3d2e5 P abe71a1ade2c4d189bd7161772a48fc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d4730368534761b1cf32cbb0a9e06f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43821 } } peers { permanent_uuid: "e0f142cb6cee44de9b506b786637fbe2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39473 } } peers { permanent_uuid: "abe71a1ade2c4d189bd7161772a48fc6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37267 } }
I20260620 02:57:08.905112 19674 raft_consensus.cc:374] T 85c101ce9285464686b12c71e3c3d2e5 P abe71a1ade2c4d189bd7161772a48fc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:08.905220 19674 raft_consensus.cc:729] T 85c101ce9285464686b12c71e3c3d2e5 P abe71a1ade2c4d189bd7161772a48fc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abe71a1ade2c4d189bd7161772a48fc6, State: Initialized, Role: FOLLOWER
I20260620 02:57:08.905876 19674 consensus_queue.cc:260] T 85c101ce9285464686b12c71e3c3d2e5 P abe71a1ade2c4d189bd7161772a48fc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d4730368534761b1cf32cbb0a9e06f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43821 } } peers { permanent_uuid: "e0f142cb6cee44de9b506b786637fbe2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39473 } } peers { permanent_uuid: "abe71a1ade2c4d189bd7161772a48fc6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37267 } }
I20260620 02:57:08.906137 19672 ts_tablet_manager.cc:1428] T 85c101ce9285464686b12c71e3c3d2e5 P 60d4730368534761b1cf32cbb0a9e06f: Time spent starting tablet: real 0.014s user 0.004s sys 0.008s
I20260620 02:57:08.907079 19523 heartbeater.cc:503] Master 127.0.97.62:34331 was elected leader, sending a full tablet report...
I20260620 02:57:08.907614 19674 ts_tablet_manager.cc:1428] T 85c101ce9285464686b12c71e3c3d2e5 P abe71a1ade2c4d189bd7161772a48fc6: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260620 02:57:08.989787 19677 raft_consensus.cc:482] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:57:08.990118 19677 raft_consensus.cc:504] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d4730368534761b1cf32cbb0a9e06f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43821 } } peers { permanent_uuid: "e0f142cb6cee44de9b506b786637fbe2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39473 } } peers { permanent_uuid: "abe71a1ade2c4d189bd7161772a48fc6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37267 } }
I20260620 02:57:08.992007 19677 leader_election.cc:288] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d4730368534761b1cf32cbb0a9e06f (127.0.97.1:43821), abe71a1ade2c4d189bd7161772a48fc6 (127.0.97.2:37267)
I20260620 02:57:09.000082 19666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:08.992659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44591 (local address 127.0.97.1:43821)
0620 02:57:08.992835 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:08.992842 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:08.992864 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:57:08.992994 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:08.992998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:08.993094 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:08.993198 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:57:08.993707 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:08.994834 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:08.997869 (+ 3035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:08.999162 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:08.999325 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:08.999431 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:08.999716 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 02:57:08.999822 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 02:57:09.000140 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:08.992442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43821 (local address 127.0.97.3:44591)
0620 02:57:08.992702 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:08.992725 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:08.992864 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:08.993236 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:08.993241 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:08.993269 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:57:08.993505 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:08.993521 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:08.995144 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:08.995148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:08.997623 (+ 2475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:08.997643 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:08.999365 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:08.999369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:08.999534 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:08.999572 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 02:57:08.999674 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 02:57:08.999825 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":3}
I20260620 02:57:09.000342 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:08.993146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58181 (local address 127.0.97.2:37267)
0620 02:57:08.993374 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:08.993382 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:08.993913 (+ 531us) server_negotiation.cc:407] Connection header received
0620 02:57:08.993999 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:08.994004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:08.994100 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:08.994200 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:57:08.994812 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:08.995588 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:08.998184 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:08.999421 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:08.999541 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:08.999606 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:08.999932 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 02:57:09.000050 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 02:57:09.000633 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:08.992963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37267 (local address 127.0.97.3:58181)
0620 02:57:08.993800 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:08.993813 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:08.993903 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:08.994231 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:08.994234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:08.994254 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:57:08.994591 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:08.994599 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:08.995836 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:08.995841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:08.997960 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:08.997970 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:08.999597 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:08.999602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:08.999775 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:08.999804 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 02:57:08.999883 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 02:57:09.000003 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"mutex_wait_us":42,"thread_start_us":492,"threads_started":1}
I20260620 02:57:09.001278 19347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85c101ce9285464686b12c71e3c3d2e5" candidate_uuid: "e0f142cb6cee44de9b506b786637fbe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d4730368534761b1cf32cbb0a9e06f" is_pre_election: true
I20260620 02:57:09.001663 19478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85c101ce9285464686b12c71e3c3d2e5" candidate_uuid: "e0f142cb6cee44de9b506b786637fbe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abe71a1ade2c4d189bd7161772a48fc6" is_pre_election: true
I20260620 02:57:09.001870 19347 raft_consensus.cc:2454] T 85c101ce9285464686b12c71e3c3d2e5 P 60d4730368534761b1cf32cbb0a9e06f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0f142cb6cee44de9b506b786637fbe2 in term 0.
I20260620 02:57:09.002154 19478 raft_consensus.cc:2454] T 85c101ce9285464686b12c71e3c3d2e5 P abe71a1ade2c4d189bd7161772a48fc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0f142cb6cee44de9b506b786637fbe2 in term 0.
I20260620 02:57:09.002635 19542 leader_election.cc:302] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d4730368534761b1cf32cbb0a9e06f, e0f142cb6cee44de9b506b786637fbe2; no voters:
I20260620 02:57:09.003273 19677 raft_consensus.cc:2781] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:57:09.003480 19677 raft_consensus.cc:482] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:57:09.003618 19677 raft_consensus.cc:3037] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:09.005914 19677 raft_consensus.cc:504] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d4730368534761b1cf32cbb0a9e06f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43821 } } peers { permanent_uuid: "e0f142cb6cee44de9b506b786637fbe2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39473 } } peers { permanent_uuid: "abe71a1ade2c4d189bd7161772a48fc6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37267 } }
I20260620 02:57:09.006587 19677 leader_election.cc:288] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2 [CANDIDATE]: Term 1 election: Requested vote from peers 60d4730368534761b1cf32cbb0a9e06f (127.0.97.1:43821), abe71a1ade2c4d189bd7161772a48fc6 (127.0.97.2:37267)
I20260620 02:57:09.007309 19478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85c101ce9285464686b12c71e3c3d2e5" candidate_uuid: "e0f142cb6cee44de9b506b786637fbe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abe71a1ade2c4d189bd7161772a48fc6"
I20260620 02:57:09.007339 19347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85c101ce9285464686b12c71e3c3d2e5" candidate_uuid: "e0f142cb6cee44de9b506b786637fbe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d4730368534761b1cf32cbb0a9e06f"
I20260620 02:57:09.007552 19478 raft_consensus.cc:3037] T 85c101ce9285464686b12c71e3c3d2e5 P abe71a1ade2c4d189bd7161772a48fc6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:09.007567 19347 raft_consensus.cc:3037] T 85c101ce9285464686b12c71e3c3d2e5 P 60d4730368534761b1cf32cbb0a9e06f [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:09.010970 19347 raft_consensus.cc:2454] T 85c101ce9285464686b12c71e3c3d2e5 P 60d4730368534761b1cf32cbb0a9e06f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0f142cb6cee44de9b506b786637fbe2 in term 1.
I20260620 02:57:09.010970 19478 raft_consensus.cc:2454] T 85c101ce9285464686b12c71e3c3d2e5 P abe71a1ade2c4d189bd7161772a48fc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0f142cb6cee44de9b506b786637fbe2 in term 1.
I20260620 02:57:09.011433 19542 leader_election.cc:302] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abe71a1ade2c4d189bd7161772a48fc6, e0f142cb6cee44de9b506b786637fbe2; no voters:
I20260620 02:57:09.011816 19677 raft_consensus.cc:2781] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:57:09.012583 19677 raft_consensus.cc:686] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2 [term 1 LEADER]: Becoming Leader. State: Replica: e0f142cb6cee44de9b506b786637fbe2, State: Running, Role: LEADER
I20260620 02:57:09.013180 19677 consensus_queue.cc:237] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d4730368534761b1cf32cbb0a9e06f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43821 } } peers { permanent_uuid: "e0f142cb6cee44de9b506b786637fbe2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39473 } } peers { permanent_uuid: "abe71a1ade2c4d189bd7161772a48fc6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37267 } }
I20260620 02:57:09.018754 19205 catalog_manager.cc:5495] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0f142cb6cee44de9b506b786637fbe2 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "e0f142cb6cee44de9b506b786637fbe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d4730368534761b1cf32cbb0a9e06f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0f142cb6cee44de9b506b786637fbe2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "abe71a1ade2c4d189bd7161772a48fc6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37267 } health_report { overall_health: UNKNOWN } } }
W20260620 02:57:09.038174 19654 tablet.cc:2369] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:57:09.065610 19668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:09.059721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39496 (local address 127.0.97.3:39473)
0620 02:57:09.059934 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:09.059938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:09.060044 (+ 106us) server_negotiation.cc:407] Connection header received
0620 02:57:09.060319 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:09.060324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:09.060438 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:09.060671 (+ 233us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:57:09.061059 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:09.061791 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:09.063273 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:09.063693 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:09.063792 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:09.064388 (+ 596us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:57:09.064907 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:57:09.065066 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:09.065350 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 02:57:09.065389 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 02:57:09.067762 19524 tablet.cc:2369] T 85c101ce9285464686b12c71e3c3d2e5 P abe71a1ade2c4d189bd7161772a48fc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:57:09.075199 19478 raft_consensus.cc:1261] T 85c101ce9285464686b12c71e3c3d2e5 P abe71a1ade2c4d189bd7161772a48fc6 [term 1 FOLLOWER]: Refusing update from remote peer e0f142cb6cee44de9b506b786637fbe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:57:09.075249 19347 raft_consensus.cc:1261] T 85c101ce9285464686b12c71e3c3d2e5 P 60d4730368534761b1cf32cbb0a9e06f [term 1 FOLLOWER]: Refusing update from remote peer e0f142cb6cee44de9b506b786637fbe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:57:09.076136 19681 consensus_queue.cc:1035] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abe71a1ade2c4d189bd7161772a48fc6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:57:09.076532 19677 consensus_queue.cc:1035] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d4730368534761b1cf32cbb0a9e06f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 02:57:09.089159 19393 tablet.cc:2369] T 85c101ce9285464686b12c71e3c3d2e5 P 60d4730368534761b1cf32cbb0a9e06f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:57:09.091420 19684 mvcc.cc:205] Tried to move back new op lower bound from 7298761642280398848 to 7298761642043351040. Current Snapshot: MvccSnapshot[applied={T|T < 7298761642280398848}]
I20260620 02:57:09.107216 19206 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:57:09.107506 19206 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:57:09.110960 19206 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 56e0e14ecc5b49b2b21e380e39d114ad: Sending DeleteTablet for 3 replicas of tablet 85c101ce9285464686b12c71e3c3d2e5
I20260620 02:57:09.112187 19588 tablet_service.cc:1514] Processing DeleteTablet for tablet 85c101ce9285464686b12c71e3c3d2e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:57:09 UTC) from {username='slave'} at 127.0.0.1:39490
I20260620 02:57:09.112166 19327 tablet_service.cc:1514] Processing DeleteTablet for tablet 85c101ce9285464686b12c71e3c3d2e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:57:09 UTC) from {username='slave'} at 127.0.0.1:38456
I20260620 02:57:09.112313 19458 tablet_service.cc:1514] Processing DeleteTablet for tablet 85c101ce9285464686b12c71e3c3d2e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:57:09 UTC) from {username='slave'} at 127.0.0.1:33762
I20260620 02:57:09.113070 19694 tablet_replica.cc:330] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2: stopping tablet replica
I20260620 02:57:09.113291 19695 tablet_replica.cc:330] T 85c101ce9285464686b12c71e3c3d2e5 P 60d4730368534761b1cf32cbb0a9e06f: stopping tablet replica
I20260620 02:57:09.113503 19694 raft_consensus.cc:2229] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:57:09.113674 19695 raft_consensus.cc:2229] T 85c101ce9285464686b12c71e3c3d2e5 P 60d4730368534761b1cf32cbb0a9e06f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:57:09.114144 19694 raft_consensus.cc:2258] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:57:09.114200 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 19265
I20260620 02:57:09.114209 19695 raft_consensus.cc:2258] T 85c101ce9285464686b12c71e3c3d2e5 P 60d4730368534761b1cf32cbb0a9e06f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:57:09.117152 19695 ts_tablet_manager.cc:1905] T 85c101ce9285464686b12c71e3c3d2e5 P 60d4730368534761b1cf32cbb0a9e06f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:57:09.117676 19696 tablet_replica.cc:330] T 85c101ce9285464686b12c71e3c3d2e5 P abe71a1ade2c4d189bd7161772a48fc6: stopping tablet replica
I20260620 02:57:09.118125 19696 raft_consensus.cc:2229] T 85c101ce9285464686b12c71e3c3d2e5 P abe71a1ade2c4d189bd7161772a48fc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:57:09.118687 19696 raft_consensus.cc:2258] T 85c101ce9285464686b12c71e3c3d2e5 P abe71a1ade2c4d189bd7161772a48fc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:57:09.119045 19694 ts_tablet_manager.cc:1905] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:57:09.124585 19666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:09.115701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38472 (local address 127.0.97.1:43821)
0620 02:57:09.115874 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:09.115880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:09.115973 (+ 93us) server_negotiation.cc:407] Connection header received
0620 02:57:09.116248 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:09.116253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:09.116345 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:09.116423 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:09.117053 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:09.118295 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:09.120406 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:09.121092 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:09.121199 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:09.122737 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:09.122752 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:09.122772 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:09.123609 (+ 837us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:09.123676 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:09.123701 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:09.123800 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:09.124049 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 02:57:09.124173 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 02:57:09.126387 19387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:57:09.126695 19694 ts_tablet_manager.cc:1918] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:57:09.126928 19694 log.cc:1196] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-2/wal/wals/85c101ce9285464686b12c71e3c3d2e5
I20260620 02:57:09.127341 19696 ts_tablet_manager.cc:1905] T 85c101ce9285464686b12c71e3c3d2e5 P abe71a1ade2c4d189bd7161772a48fc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:57:09.127655 19694 ts_tablet_manager.cc:1939] T 85c101ce9285464686b12c71e3c3d2e5 P e0f142cb6cee44de9b506b786637fbe2: Deleting consensus metadata
I20260620 02:57:09.129611 19191 catalog_manager.cc:4841] TS e0f142cb6cee44de9b506b786637fbe2 (127.0.97.3:39473): tablet 85c101ce9285464686b12c71e3c3d2e5 (table test-table [id=d7b5302ef192408f8096fb9ba50c2be6]) successfully deleted
I20260620 02:57:09.135547 19696 ts_tablet_manager.cc:1918] T 85c101ce9285464686b12c71e3c3d2e5 P abe71a1ade2c4d189bd7161772a48fc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:57:09.135720 19696 log.cc:1196] T 85c101ce9285464686b12c71e3c3d2e5 P abe71a1ade2c4d189bd7161772a48fc6: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-1/wal/wals/85c101ce9285464686b12c71e3c3d2e5
I20260620 02:57:09.136220 19696 ts_tablet_manager.cc:1939] T 85c101ce9285464686b12c71e3c3d2e5 P abe71a1ade2c4d189bd7161772a48fc6: Deleting consensus metadata
I20260620 02:57:09.137737 19191 catalog_manager.cc:4841] TS abe71a1ade2c4d189bd7161772a48fc6 (127.0.97.2:37267): tablet 85c101ce9285464686b12c71e3c3d2e5 (table test-table [id=d7b5302ef192408f8096fb9ba50c2be6]) successfully deleted
I20260620 02:57:09.242292 19695 ts_tablet_manager.cc:1918] T 85c101ce9285464686b12c71e3c3d2e5 P 60d4730368534761b1cf32cbb0a9e06f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:57:09.242703 19695 log.cc:1196] T 85c101ce9285464686b12c71e3c3d2e5 P 60d4730368534761b1cf32cbb0a9e06f: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781923914418765-388-0/minicluster-data/ts-0/wal/wals/85c101ce9285464686b12c71e3c3d2e5
I20260620 02:57:09.243399 19695 ts_tablet_manager.cc:1939] T 85c101ce9285464686b12c71e3c3d2e5 P 60d4730368534761b1cf32cbb0a9e06f: Deleting consensus metadata
I20260620 02:57:09.243402 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 19265
W20260620 02:57:09.268229 19191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:43821 (error 108)
I20260620 02:57:09.269053 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 19396
I20260620 02:57:09.270768 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:09.270089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43821 (local address 127.0.0.1:38476)
0620 02:57:09.270333 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:09.270471 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
W20260620 02:57:09.271039 19191 catalog_manager.cc:4584] TS 60d4730368534761b1cf32cbb0a9e06f (127.0.97.1:43821): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 85c101ce9285464686b12c71e3c3d2e5: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43821: connect: Connection refused (error 111)
I20260620 02:57:09.277278 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:09.269980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33774 (local address 127.0.97.2:37267)
0620 02:57:09.270168 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:09.270174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:09.270195 (+ 21us) server_negotiation.cc:407] Connection header received
0620 02:57:09.270333 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:09.270340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:09.270485 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:09.270704 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:09.271066 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:09.271958 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:09.273875 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:09.274459 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:09.274639 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:09.275823 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:09.275832 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:09.275847 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:09.276450 (+ 603us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:09.276507 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:09.276527 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:09.276620 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:09.276776 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 02:57:09.276873 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:57:09.277989 19518 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:57:09.324949 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:09.324367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43821 (local address 127.0.0.1:38490)
0620 02:57:09.324628 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:09.324744 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 02:57:09.326195 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:09.325773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43821 (local address 127.0.0.1:38492)
0620 02:57:09.325944 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:09.326031 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 02:57:09.369117 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 19396
I20260620 02:57:09.386873 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:09.386340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43821 (local address 127.0.0.1:38508)
0620 02:57:09.386564 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:09.386674 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 02:57:09.386870 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 19526
I20260620 02:57:09.388356 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:09.387947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43821 (local address 127.0.0.1:38514)
0620 02:57:09.388094 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:09.388207 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 02:57:09.396056 19668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:09.387659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39510 (local address 127.0.97.3:39473)
0620 02:57:09.387847 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:09.387853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:09.387920 (+ 67us) server_negotiation.cc:407] Connection header received
0620 02:57:09.388194 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:09.388201 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:09.388301 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:09.388443 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:09.389043 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:09.390240 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:09.392476 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:09.393112 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:09.393238 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:09.394537 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:09.394547 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:09.394562 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:09.395224 (+ 662us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:09.395277 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:09.395304 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:09.395399 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:09.395609 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 02:57:09.395687 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 02:57:09.396871 19648 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:57:09.469120 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:09.468620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43821 (local address 127.0.0.1:38528)
0620 02:57:09.468844 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:09.468945 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 02:57:09.470191 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:09.469817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43821 (local address 127.0.0.1:38542)
0620 02:57:09.470009 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:09.470089 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260620 02:57:09.516206 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 19526
I20260620 02:57:09.535564 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 19175
I20260620 02:57:09.536801 19236 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:57:09.647123 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:09.646444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43821 (local address 127.0.0.1:38556)
0620 02:57:09.646749 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:09.646869 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 02:57:09.647269 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 19175
2026-06-20T02:57:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2929 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:57:09 dist-test-slave-8p8c krb5kdc[19705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:57:09 dist-test-slave-8p8c krb5kdc[19705](info): set up 2 sockets
Jun 20 02:57:09 dist-test-slave-8p8c krb5kdc[19705](info): commencing operation
krb5kdc: starting...
W20260620 02:57:10.758664 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.008s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:57:10 dist-test-slave-8p8c krb5kdc[19705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:57:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:57:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:57:11.014032 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44231
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44231
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:57:11.360641 19718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:11.360991 19718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:11.361145 19718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:11.368515 19718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:57:11.368662 19718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:11.368772 19718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:11.368863 19718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:57:11.368942 19718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:57:11.379014 19718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:11.380333 19718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:11.381680 19718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:11.393571 19723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:11.393582 19724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:11.393599 19726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:11.394232 19718 server_base.cc:1034] running on GCE node
I20260620 02:57:11.395082 19718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:11.396436 19718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:11.397666 19718 hybrid_clock.cc:648] HybridClock initialized: now 1781924231397608 us; error 73 us; skew 500 ppm
I20260620 02:57:11.397997 19718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:57:11 dist-test-slave-8p8c krb5kdc[19705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:57:11.401006 19718 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:57:11.402557 19718 webserver.cc:460] Webserver started at http://127.0.97.62:37077/ using document root <none> and password file <none>
I20260620 02:57:11.403230 19718 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:11.403366 19718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:11.403683 19718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:11.406700 19718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "cd7f01ccf152411aba47dfa0bd884c3f"
format_stamp: "Formatted at 2026-06-20 02:57:11 on dist-test-slave-8p8c"
server_key: "a8ffaee3911b223cb1e64e869cc3ed96"
server_key_iv: "5ecca6901f058d7ec7158294d73eae1b"
server_key_version: "encryptionkey@0"
I20260620 02:57:11.407450 19718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "cd7f01ccf152411aba47dfa0bd884c3f"
format_stamp: "Formatted at 2026-06-20 02:57:11 on dist-test-slave-8p8c"
server_key: "a8ffaee3911b223cb1e64e869cc3ed96"
server_key_iv: "5ecca6901f058d7ec7158294d73eae1b"
server_key_version: "encryptionkey@0"
I20260620 02:57:11.412634 19718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:57:11.416424 19733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:11.417870 19718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:57:11.418079 19718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "cd7f01ccf152411aba47dfa0bd884c3f"
format_stamp: "Formatted at 2026-06-20 02:57:11 on dist-test-slave-8p8c"
server_key: "a8ffaee3911b223cb1e64e869cc3ed96"
server_key_iv: "5ecca6901f058d7ec7158294d73eae1b"
server_key_version: "encryptionkey@0"
I20260620 02:57:11.418274 19718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:11.428979 19718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:11.433048 19718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:11.433326 19718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:11.451539 19718 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44231
I20260620 02:57:11.451570 19784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44231 every 8 connection(s)
I20260620 02:57:11.452941 19718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:57:11.456888 19785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:11.458503 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 19718
I20260620 02:57:11.458694 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:57:11.459246 388 external_mini_cluster.cc:1428] Setting key 82d584c9bb3108169bcc64acb6e9c7bc
I20260620 02:57:11.471946 19785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7f01ccf152411aba47dfa0bd884c3f: Bootstrap starting.
Jun 20 02:57:11 dist-test-slave-8p8c krb5kdc[19705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:57:11.475646 19785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd7f01ccf152411aba47dfa0bd884c3f: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:11.476696 19785 log.cc:824] T 00000000000000000000000000000000 P cd7f01ccf152411aba47dfa0bd884c3f: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:11.479812 19785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7f01ccf152411aba47dfa0bd884c3f: No bootstrap required, opened a new log
I20260620 02:57:11.483865 19788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:11.461215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33270 (local address 127.0.97.62:44231)
0620 02:57:11.462100 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:11.462112 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:11.462155 (+ 43us) server_negotiation.cc:407] Connection header received
0620 02:57:11.463932 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:11.463966 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:11.464383 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:11.464721 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:11.466599 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:11.468091 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:11.469925 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:11.470709 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:11.470840 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:11.475837 (+ 4997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:11.475847 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:11.475864 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:11.475906 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:11.479554 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:11.480341 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:11.480348 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:11.480355 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:11.480530 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:11.480924 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:11.480931 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:11.480944 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:11.481485 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:11.481833 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:11.482292 (+ 459us) server_negotiation.cc:299] Negotiation successful
0620 02:57:11.482796 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":448,"threads_started":1}
I20260620 02:57:11.487697 19785 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd7f01ccf152411aba47dfa0bd884c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7f01ccf152411aba47dfa0bd884c3f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44231 } }
I20260620 02:57:11.488052 19785 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd7f01ccf152411aba47dfa0bd884c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:11.488157 19785 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd7f01ccf152411aba47dfa0bd884c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7f01ccf152411aba47dfa0bd884c3f, State: Initialized, Role: FOLLOWER
I20260620 02:57:11.488823 19785 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd7f01ccf152411aba47dfa0bd884c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7f01ccf152411aba47dfa0bd884c3f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44231 } }
I20260620 02:57:11.489122 19785 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd7f01ccf152411aba47dfa0bd884c3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:57:11.489250 19785 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd7f01ccf152411aba47dfa0bd884c3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:57:11.489418 19785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd7f01ccf152411aba47dfa0bd884c3f [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:11.491647 19785 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd7f01ccf152411aba47dfa0bd884c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7f01ccf152411aba47dfa0bd884c3f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44231 } }
I20260620 02:57:11.492066 19785 leader_election.cc:302] T 00000000000000000000000000000000 P cd7f01ccf152411aba47dfa0bd884c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd7f01ccf152411aba47dfa0bd884c3f; no voters:
I20260620 02:57:11.492729 19785 leader_election.cc:288] T 00000000000000000000000000000000 P cd7f01ccf152411aba47dfa0bd884c3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:57:11.492857 19790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd7f01ccf152411aba47dfa0bd884c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:57:11.493880 19790 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd7f01ccf152411aba47dfa0bd884c3f [term 1 LEADER]: Becoming Leader. State: Replica: cd7f01ccf152411aba47dfa0bd884c3f, State: Running, Role: LEADER
I20260620 02:57:11.494436 19790 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd7f01ccf152411aba47dfa0bd884c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7f01ccf152411aba47dfa0bd884c3f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44231 } }
I20260620 02:57:11.494901 19785 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd7f01ccf152411aba47dfa0bd884c3f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:57:11.497059 19791 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7f01ccf152411aba47dfa0bd884c3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd7f01ccf152411aba47dfa0bd884c3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7f01ccf152411aba47dfa0bd884c3f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44231 } } }
I20260620 02:57:11.497498 19791 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7f01ccf152411aba47dfa0bd884c3f [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:11.498407 19792 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7f01ccf152411aba47dfa0bd884c3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd7f01ccf152411aba47dfa0bd884c3f. Latest consensus state: current_term: 1 leader_uuid: "cd7f01ccf152411aba47dfa0bd884c3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7f01ccf152411aba47dfa0bd884c3f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44231 } } }
I20260620 02:57:11.498699 19792 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7f01ccf152411aba47dfa0bd884c3f [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:11.499429 19799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:57:11.504873 19799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:57:11.513278 19799 catalog_manager.cc:1347] Generated new cluster ID: 340f0de8816f49428353a2cbbbc36510
I20260620 02:57:11.513392 19799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:57:11.526465 19799 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:57:11.528438 19799 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:57:11.548174 19799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd7f01ccf152411aba47dfa0bd884c3f: Generated new TSK 0
I20260620 02:57:11.549113 19799 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 02:57:11.623835 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44231
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:57:11.973021 19811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:11.973358 19811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:11.973521 19811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:11.981408 19811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:11.981559 19811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:11.981729 19811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:57:11.991888 19811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:11.993270 19811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:11.994748 19811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:12.008313 19817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:12.008296 19816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:12.008836 19819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:12.009102 19811 server_base.cc:1034] running on GCE node
I20260620 02:57:12.009698 19811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:12.010710 19811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:12.011942 19811 hybrid_clock.cc:648] HybridClock initialized: now 1781924232011892 us; error 30 us; skew 500 ppm
I20260620 02:57:12.012277 19811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:57:12 dist-test-slave-8p8c krb5kdc[19705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:57:12.015457 19811 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 02:57:12.016847 19811 webserver.cc:460] Webserver started at http://127.0.97.1:38507/ using document root <none> and password file <none>
I20260620 02:57:12.017485 19811 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:12.017590 19811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:12.017843 19811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:12.020714 19811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-0/data/instance:
uuid: "fd9ff8462e4446458f52aa36793a8d32"
format_stamp: "Formatted at 2026-06-20 02:57:12 on dist-test-slave-8p8c"
server_key: "3eab8c344c27c3c9216018ef9746e75f"
server_key_iv: "aa35c62d107125637f2238b897bc7a7b"
server_key_version: "encryptionkey@0"
I20260620 02:57:12.021520 19811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-0/wal/instance:
uuid: "fd9ff8462e4446458f52aa36793a8d32"
format_stamp: "Formatted at 2026-06-20 02:57:12 on dist-test-slave-8p8c"
server_key: "3eab8c344c27c3c9216018ef9746e75f"
server_key_iv: "aa35c62d107125637f2238b897bc7a7b"
server_key_version: "encryptionkey@0"
I20260620 02:57:12.026729 19811 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 02:57:12.030056 19826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:12.031530 19811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:57:12.031670 19811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-0/wal
uuid: "fd9ff8462e4446458f52aa36793a8d32"
format_stamp: "Formatted at 2026-06-20 02:57:12 on dist-test-slave-8p8c"
server_key: "3eab8c344c27c3c9216018ef9746e75f"
server_key_iv: "aa35c62d107125637f2238b897bc7a7b"
server_key_version: "encryptionkey@0"
I20260620 02:57:12.031855 19811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:12.064334 19811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:12.068846 19811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:12.069105 19811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:12.070096 19811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:57:12.071480 19811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:57:12.071552 19811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:12.071635 19811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:57:12.071686 19811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:12.105293 19811 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46209
I20260620 02:57:12.105382 19938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46209 every 8 connection(s)
I20260620 02:57:12.106675 19811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:57:12.112211 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 19811
I20260620 02:57:12.112386 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:57:12.112936 388 external_mini_cluster.cc:1428] Setting key 1481a61e660de9e30b4a32c5bd6ccd75
Jun 20 02:57:12 dist-test-slave-8p8c krb5kdc[19705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:57:12.128748 19941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:12.109414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55803 (local address 127.0.97.62:44231)
0620 02:57:12.109885 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:12.109890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:12.110878 (+ 988us) server_negotiation.cc:407] Connection header received
0620 02:57:12.113191 (+ 2313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:12.113195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:12.113278 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:12.113375 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:12.115254 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:12.115977 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:12.118517 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:12.118905 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:12.119014 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:12.123237 (+ 4223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:12.123242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:12.123245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:12.123276 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:12.125796 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:12.126516 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:12.126525 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:12.126527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:12.126642 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:12.127063 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:12.127068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:12.127071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:12.127320 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:12.127486 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:12.128173 (+ 687us) server_negotiation.cc:299] Negotiation successful
0620 02:57:12.128395 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":238,"threads_started":1}
I20260620 02:57:12.129714 19942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:12.110008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44231 (local address 127.0.97.1:55803)
0620 02:57:12.110716 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:12.110756 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:12.112871 (+ 2115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:12.113634 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:12.113643 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:12.114315 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:57:12.115074 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:12.115100 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:12.116146 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:12.116155 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:12.118336 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:12.118353 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:12.119140 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:12.119149 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:12.119463 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:12.120198 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:57:12.120233 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:57:12.123016 (+ 2783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:57:12.125927 (+ 2911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:57:12.125935 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:57:12.125950 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:57:12.126355 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:57:12.126755 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:57:12.126761 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:57:12.126764 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:57:12.126952 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:57:12.127450 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:57:12.127459 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:57:12.127790 (+ 331us) client_negotiation.cc:769] Sending connection context
0620 02:57:12.128060 (+ 270us) client_negotiation.cc:240] Negotiation successful
0620 02:57:12.128584 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":305,"threads_started":1}
I20260620 02:57:12.131366 19939 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44231
I20260620 02:57:12.131701 19939 heartbeater.cc:464] Registering TS with master...
I20260620 02:57:12.132560 19939 heartbeater.cc:511] Master 127.0.97.62:44231 requested a full tablet report, sending...
I20260620 02:57:12.135118 19750 ts_manager.cc:194] Registered new tserver with Master: fd9ff8462e4446458f52aa36793a8d32 (127.0.97.1:46209)
I20260620 02:57:12.137487 19750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55803
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 02:57:12.171648 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44231
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:57:12.499805 19945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:12.500115 19945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:12.500272 19945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:12.507766 19945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:12.507885 19945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:12.508065 19945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:57:12.518039 19945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:12.519448 19945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:12.520812 19945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:12.534965 19950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:12.535020 19951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:12.535045 19953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:12.535406 19945 server_base.cc:1034] running on GCE node
I20260620 02:57:12.536316 19945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:12.537241 19945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:12.538501 19945 hybrid_clock.cc:648] HybridClock initialized: now 1781924232538436 us; error 44 us; skew 500 ppm
I20260620 02:57:12.538852 19945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:57:12 dist-test-slave-8p8c krb5kdc[19705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:57:12.541963 19945 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 02:57:12.543629 19945 webserver.cc:460] Webserver started at http://127.0.97.2:43389/ using document root <none> and password file <none>
I20260620 02:57:12.544332 19945 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:12.544457 19945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:12.544766 19945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:12.547767 19945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-1/data/instance:
uuid: "496e672ea39944f180f7b6509c0e5aa1"
format_stamp: "Formatted at 2026-06-20 02:57:12 on dist-test-slave-8p8c"
server_key: "a1cfc47b6007aa38ca903cbe7d0eac18"
server_key_iv: "249deaca9c893cac961b66b91b4b45a8"
server_key_version: "encryptionkey@0"
I20260620 02:57:12.548663 19945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-1/wal/instance:
uuid: "496e672ea39944f180f7b6509c0e5aa1"
format_stamp: "Formatted at 2026-06-20 02:57:12 on dist-test-slave-8p8c"
server_key: "a1cfc47b6007aa38ca903cbe7d0eac18"
server_key_iv: "249deaca9c893cac961b66b91b4b45a8"
server_key_version: "encryptionkey@0"
I20260620 02:57:12.553776 19945 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:57:12.557626 19960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:12.559324 19945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:57:12.559484 19945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-1/wal
uuid: "496e672ea39944f180f7b6509c0e5aa1"
format_stamp: "Formatted at 2026-06-20 02:57:12 on dist-test-slave-8p8c"
server_key: "a1cfc47b6007aa38ca903cbe7d0eac18"
server_key_iv: "249deaca9c893cac961b66b91b4b45a8"
server_key_version: "encryptionkey@0"
I20260620 02:57:12.559690 19945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:12.583149 19945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:12.588125 19945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:12.588433 19945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:12.589525 19945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:57:12.590998 19945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:57:12.591104 19945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:12.591241 19945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:57:12.591310 19945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:12.623643 19945 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36347
I20260620 02:57:12.623726 20072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36347 every 8 connection(s)
I20260620 02:57:12.624936 19945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:57:12.626601 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 19945
I20260620 02:57:12.626785 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:57:12.627300 388 external_mini_cluster.cc:1428] Setting key 8be5ee514a2d8012e0ba169457248632
Jun 20 02:57:12 dist-test-slave-8p8c krb5kdc[19705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:57:12.647616 19941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:12.627761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50937 (local address 127.0.97.62:44231)
0620 02:57:12.627960 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:12.627969 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:12.629308 (+ 1339us) server_negotiation.cc:407] Connection header received
0620 02:57:12.631453 (+ 2145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:12.631462 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:12.631554 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:12.631668 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:12.633250 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:12.634429 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:12.636492 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:12.637190 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:12.637333 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:12.641503 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:12.641509 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:12.641512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:12.641550 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:12.644423 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:12.645198 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:12.645204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:12.645207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:12.645339 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:12.645769 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:12.645775 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:12.645778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:12.646070 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:12.646236 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:12.647031 (+ 795us) server_negotiation.cc:299] Negotiation successful
0620 02:57:12.647267 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 02:57:12.648764 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:12.628315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44231 (local address 127.0.97.2:50937)
0620 02:57:12.629138 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:12.629190 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:12.631178 (+ 1988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:12.631880 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:12.631889 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:12.632502 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:57:12.633076 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:12.633094 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:12.634618 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:12.634622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:12.636345 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:12.636359 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:12.637443 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:12.637452 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:12.637698 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:12.638574 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:57:12.638605 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:57:12.641246 (+ 2641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:57:12.644594 (+ 3348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:57:12.644603 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:57:12.644622 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:57:12.645070 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:57:12.645450 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:57:12.645454 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:57:12.645458 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:57:12.645656 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:57:12.646197 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:57:12.646205 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:57:12.646627 (+ 422us) client_negotiation.cc:769] Sending connection context
0620 02:57:12.646923 (+ 296us) client_negotiation.cc:240] Negotiation successful
0620 02:57:12.647510 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":350,"threads_started":1}
I20260620 02:57:12.650534 20073 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44231
I20260620 02:57:12.650852 20073 heartbeater.cc:464] Registering TS with master...
I20260620 02:57:12.651604 20073 heartbeater.cc:511] Master 127.0.97.62:44231 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 02:57:12.653373 19750 ts_manager.cc:194] Registered new tserver with Master: 496e672ea39944f180f7b6509c0e5aa1 (127.0.97.2:36347)
I20260620 02:57:12.654279 19750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 02:57:12.685670 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44231
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:57:13.047442 20078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:13.047755 20078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:13.047902 20078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:13.055842 20078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:13.055967 20078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:13.056149 20078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:57:13.066337 20078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:13.067725 20078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:13.069067 20078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:13.083103 20083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:13.083163 20084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:13.083194 20086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:13.083913 20078 server_base.cc:1034] running on GCE node
I20260620 02:57:13.084533 20078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:13.085484 20078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:13.086671 20078 hybrid_clock.cc:648] HybridClock initialized: now 1781924233086613 us; error 43 us; skew 500 ppm
I20260620 02:57:13.086984 20078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:57:13 dist-test-slave-8p8c krb5kdc[19705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:57:13.090231 20078 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 02:57:13.091673 20078 webserver.cc:460] Webserver started at http://127.0.97.3:33689/ using document root <none> and password file <none>
I20260620 02:57:13.092376 20078 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:13.092515 20078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:13.092826 20078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:13.095846 20078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-2/data/instance:
uuid: "c011e6a2271341dc9cf323fefacdd776"
format_stamp: "Formatted at 2026-06-20 02:57:13 on dist-test-slave-8p8c"
server_key: "f110c2a80b385f665be7769c30c44aea"
server_key_iv: "4c727dfdd44ad101823713c9b5396575"
server_key_version: "encryptionkey@0"
I20260620 02:57:13.096657 20078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c011e6a2271341dc9cf323fefacdd776"
format_stamp: "Formatted at 2026-06-20 02:57:13 on dist-test-slave-8p8c"
server_key: "f110c2a80b385f665be7769c30c44aea"
server_key_iv: "4c727dfdd44ad101823713c9b5396575"
server_key_version: "encryptionkey@0"
I20260620 02:57:13.101714 20078 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:57:13.105266 20093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:13.106832 20078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:57:13.107039 20078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-2/wal
uuid: "c011e6a2271341dc9cf323fefacdd776"
format_stamp: "Formatted at 2026-06-20 02:57:13 on dist-test-slave-8p8c"
server_key: "f110c2a80b385f665be7769c30c44aea"
server_key_iv: "4c727dfdd44ad101823713c9b5396575"
server_key_version: "encryptionkey@0"
I20260620 02:57:13.107259 20078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:13.125466 20078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:13.130191 20078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:13.130558 20078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:13.131563 20078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:57:13.132958 20078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:57:13.133067 20078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:13.133219 20078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:57:13.133293 20078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:13.141160 19939 heartbeater.cc:503] Master 127.0.97.62:44231 was elected leader, sending a full tablet report...
I20260620 02:57:13.168944 20078 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44111
I20260620 02:57:13.168991 20205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44111 every 8 connection(s)
I20260620 02:57:13.170222 20078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:57:13.171828 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 20078
I20260620 02:57:13.172001 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:57:13.172416 388 external_mini_cluster.cc:1428] Setting key db3ae8822112754c71cd5cb61aee60c0
Jun 20 02:57:13 dist-test-slave-8p8c krb5kdc[19705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:57:13.191085 20208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:13.173015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38707 (local address 127.0.97.62:44231)
0620 02:57:13.173518 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:13.173524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:13.174709 (+ 1185us) server_negotiation.cc:407] Connection header received
0620 02:57:13.176596 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:13.176600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:13.176668 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:13.176769 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:13.178426 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:13.179267 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:13.180901 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:13.181333 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:13.181430 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:13.184916 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:13.184921 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:13.184925 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:13.184958 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:13.188117 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:13.188852 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:13.188857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:13.188860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:13.188993 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:13.189382 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:13.189385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:13.189387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:13.189643 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:13.189751 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:13.190655 (+ 904us) server_negotiation.cc:299] Negotiation successful
0620 02:57:13.190817 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":241,"threads_started":1}
I20260620 02:57:13.192175 20209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:13.173641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44231 (local address 127.0.97.3:38707)
0620 02:57:13.174541 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:13.174597 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:13.176326 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:13.177003 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:13.177011 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:13.177624 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:57:13.178211 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:13.178228 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:13.179429 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:13.179434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:13.180758 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:13.180771 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:13.181550 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:13.181556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:13.181757 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:13.182508 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:57:13.182541 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:57:13.184694 (+ 2153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:57:13.188247 (+ 3553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:57:13.188255 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:57:13.188268 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:57:13.188742 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:57:13.189088 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:57:13.189091 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:57:13.189094 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:57:13.189289 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:57:13.189801 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:57:13.189807 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:57:13.190226 (+ 419us) client_negotiation.cc:769] Sending connection context
0620 02:57:13.190585 (+ 359us) client_negotiation.cc:240] Negotiation successful
0620 02:57:13.191012 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":374,"threads_started":1}
I20260620 02:57:13.193791 20206 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44231
I20260620 02:57:13.194079 20206 heartbeater.cc:464] Registering TS with master...
I20260620 02:57:13.194919 20206 heartbeater.cc:511] Master 127.0.97.62:44231 requested a full tablet report, sending...
I20260620 02:57:13.196502 19750 ts_manager.cc:194] Registered new tserver with Master: c011e6a2271341dc9cf323fefacdd776 (127.0.97.3:44111)
I20260620 02:57:13.197393 19750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38707
I20260620 02:57:13.200229 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:57:13.216873 20208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:13.204251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35984 (local address 127.0.97.62:44231)
0620 02:57:13.204413 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:13.204417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:13.204987 (+ 570us) server_negotiation.cc:407] Connection header received
0620 02:57:13.205148 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:13.205152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:13.205221 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:13.205325 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:13.206723 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:13.207491 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:13.208790 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:13.209233 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:13.209333 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:13.211763 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:13.211768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:13.211772 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:13.211805 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:13.214625 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:13.215290 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:13.215294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:13.215296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:13.215410 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:13.215720 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:13.215723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:13.215725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:13.216024 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:13.216197 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:13.216486 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 02:57:13.216649 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 02:57:13.220870 19750 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:57:13.227985 19750 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:57:13.251905 20220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:13.240155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46209 (local address 127.0.0.1:52880)
0620 02:57:13.241059 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:13.241110 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:13.241337 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:13.243742 (+ 2405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:13.243753 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:13.243803 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:57:13.244405 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:13.244423 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:13.246802 (+ 2379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:13.246807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:13.249023 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:13.249034 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:13.250714 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:13.250721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:13.250947 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:13.251003 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 02:57:13.251157 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 02:57:13.251371 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"mutex_wait_us":34,"thread_start_us":4407,"threads_started":1}
I20260620 02:57:13.252807 20223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:13.240168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44111 (local address 127.0.0.1:32918)
0620 02:57:13.242350 (+ 2182us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:13.242398 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:13.242541 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:13.243831 (+ 1290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:13.243840 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:13.243885 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:57:13.245001 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:13.245013 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:13.246955 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:13.246961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:13.249650 (+ 2689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:13.249665 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:13.251732 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:13.251739 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:13.251927 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:13.251959 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 02:57:13.252064 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 02:57:13.252197 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2050,"mutex_wait_us":50,"thread_start_us":469,"threads_started":1}
I20260620 02:57:13.253082 20222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:13.240909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52880 (local address 127.0.97.1:46209)
0620 02:57:13.242949 (+ 2040us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:13.242962 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:13.242997 (+ 35us) server_negotiation.cc:407] Connection header received
0620 02:57:13.243117 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:13.243128 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:13.243436 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:13.244660 (+ 1224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:57:13.244929 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:13.246643 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:13.249179 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:13.250445 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:13.250639 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:13.250888 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:13.251218 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 02:57:13.252724 (+ 1506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1836,"thread_start_us":1682,"threads_started":1}
I20260620 02:57:13.253371 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:13.240514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32918 (local address 127.0.97.3:44111)
0620 02:57:13.242062 (+ 1548us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:13.242073 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:13.242696 (+ 623us) server_negotiation.cc:407] Connection header received
0620 02:57:13.242838 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:13.242851 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:13.243204 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:13.243458 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:57:13.245239 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:13.246804 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:13.249825 (+ 3021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:13.251211 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:13.251338 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:13.251580 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:13.252783 (+ 1203us) server_negotiation.cc:299] Negotiation successful
0620 02:57:13.252989 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1346,"thread_start_us":1181,"threads_started":1}
I20260620 02:57:13.254180 20224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:13.242239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41952 (local address 127.0.97.2:36347)
0620 02:57:13.244499 (+ 2260us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:13.244510 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:13.245220 (+ 710us) server_negotiation.cc:407] Connection header received
0620 02:57:13.245331 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:13.245343 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:13.245611 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:13.245814 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:57:13.246322 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:13.248121 (+ 1799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:13.250559 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:13.252180 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:13.252306 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:13.252531 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:13.253658 (+ 1127us) server_negotiation.cc:299] Negotiation successful
0620 02:57:13.253849 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2067,"thread_start_us":1961,"threads_started":1}
I20260620 02:57:13.254627 20219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:13.241702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36347 (local address 127.0.0.1:41952)
0620 02:57:13.244901 (+ 3199us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:13.244935 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:13.245077 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:13.245797 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:13.245802 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:13.245833 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:57:13.246159 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:13.246168 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:13.248317 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:13.248323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:13.250353 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:13.250409 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:13.252705 (+ 2296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:13.252713 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:13.252895 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:13.252944 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 02:57:13.253039 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 02:57:13.253187 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3057,"thread_start_us":722,"threads_started":1}
I20260620 02:57:13.257392 19874 tablet_service.cc:1467] Processing CreateTablet for tablet 4badb414158f4efa9e36bcbcc2c5965a (DEFAULT_TABLE table=test-table [id=bc15692b71c645a386f498b4094fd18e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:57:13.257774 20141 tablet_service.cc:1467] Processing CreateTablet for tablet 4badb414158f4efa9e36bcbcc2c5965a (DEFAULT_TABLE table=test-table [id=bc15692b71c645a386f498b4094fd18e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:57:13.258074 20008 tablet_service.cc:1467] Processing CreateTablet for tablet 4badb414158f4efa9e36bcbcc2c5965a (DEFAULT_TABLE table=test-table [id=bc15692b71c645a386f498b4094fd18e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:57:13.258769 19874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4badb414158f4efa9e36bcbcc2c5965a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:13.259239 20141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4badb414158f4efa9e36bcbcc2c5965a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:13.259294 20008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4badb414158f4efa9e36bcbcc2c5965a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:13.270180 20225 tablet_bootstrap.cc:492] T 4badb414158f4efa9e36bcbcc2c5965a P fd9ff8462e4446458f52aa36793a8d32: Bootstrap starting.
I20260620 02:57:13.273433 20226 tablet_bootstrap.cc:492] T 4badb414158f4efa9e36bcbcc2c5965a P 496e672ea39944f180f7b6509c0e5aa1: Bootstrap starting.
I20260620 02:57:13.273464 20225 tablet_bootstrap.cc:654] T 4badb414158f4efa9e36bcbcc2c5965a P fd9ff8462e4446458f52aa36793a8d32: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:13.274879 20227 tablet_bootstrap.cc:492] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776: Bootstrap starting.
I20260620 02:57:13.275513 20225 log.cc:824] T 4badb414158f4efa9e36bcbcc2c5965a P fd9ff8462e4446458f52aa36793a8d32: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:13.278012 20226 tablet_bootstrap.cc:654] T 4badb414158f4efa9e36bcbcc2c5965a P 496e672ea39944f180f7b6509c0e5aa1: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:13.278493 20227 tablet_bootstrap.cc:654] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:13.279022 20225 tablet_bootstrap.cc:492] T 4badb414158f4efa9e36bcbcc2c5965a P fd9ff8462e4446458f52aa36793a8d32: No bootstrap required, opened a new log
I20260620 02:57:13.279222 20226 log.cc:824] T 4badb414158f4efa9e36bcbcc2c5965a P 496e672ea39944f180f7b6509c0e5aa1: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:13.279301 20225 ts_tablet_manager.cc:1397] T 4badb414158f4efa9e36bcbcc2c5965a P fd9ff8462e4446458f52aa36793a8d32: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 02:57:13.279664 20227 log.cc:824] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:13.282246 20226 tablet_bootstrap.cc:492] T 4badb414158f4efa9e36bcbcc2c5965a P 496e672ea39944f180f7b6509c0e5aa1: No bootstrap required, opened a new log
I20260620 02:57:13.282289 20227 tablet_bootstrap.cc:492] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776: No bootstrap required, opened a new log
I20260620 02:57:13.282505 20226 ts_tablet_manager.cc:1397] T 4badb414158f4efa9e36bcbcc2c5965a P 496e672ea39944f180f7b6509c0e5aa1: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 02:57:13.282588 20227 ts_tablet_manager.cc:1397] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.006s
I20260620 02:57:13.289881 20227 raft_consensus.cc:348] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c011e6a2271341dc9cf323fefacdd776" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44111 } } peers { permanent_uuid: "fd9ff8462e4446458f52aa36793a8d32" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46209 } } peers { permanent_uuid: "496e672ea39944f180f7b6509c0e5aa1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36347 } }
I20260620 02:57:13.289903 20225 raft_consensus.cc:348] T 4badb414158f4efa9e36bcbcc2c5965a P fd9ff8462e4446458f52aa36793a8d32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c011e6a2271341dc9cf323fefacdd776" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44111 } } peers { permanent_uuid: "fd9ff8462e4446458f52aa36793a8d32" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46209 } } peers { permanent_uuid: "496e672ea39944f180f7b6509c0e5aa1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36347 } }
I20260620 02:57:13.290257 20227 raft_consensus.cc:374] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:13.290321 20225 raft_consensus.cc:374] T 4badb414158f4efa9e36bcbcc2c5965a P fd9ff8462e4446458f52aa36793a8d32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:13.290428 20225 raft_consensus.cc:729] T 4badb414158f4efa9e36bcbcc2c5965a P fd9ff8462e4446458f52aa36793a8d32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd9ff8462e4446458f52aa36793a8d32, State: Initialized, Role: FOLLOWER
I20260620 02:57:13.290405 20227 raft_consensus.cc:729] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c011e6a2271341dc9cf323fefacdd776, State: Initialized, Role: FOLLOWER
I20260620 02:57:13.291038 20227 consensus_queue.cc:260] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c011e6a2271341dc9cf323fefacdd776" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44111 } } peers { permanent_uuid: "fd9ff8462e4446458f52aa36793a8d32" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46209 } } peers { permanent_uuid: "496e672ea39944f180f7b6509c0e5aa1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36347 } }
I20260620 02:57:13.291035 20225 consensus_queue.cc:260] T 4badb414158f4efa9e36bcbcc2c5965a P fd9ff8462e4446458f52aa36793a8d32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c011e6a2271341dc9cf323fefacdd776" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44111 } } peers { permanent_uuid: "fd9ff8462e4446458f52aa36793a8d32" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46209 } } peers { permanent_uuid: "496e672ea39944f180f7b6509c0e5aa1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36347 } }
I20260620 02:57:13.292171 20206 heartbeater.cc:503] Master 127.0.97.62:44231 was elected leader, sending a full tablet report...
I20260620 02:57:13.292951 20227 ts_tablet_manager.cc:1428] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260620 02:57:13.292798 20226 raft_consensus.cc:348] T 4badb414158f4efa9e36bcbcc2c5965a P 496e672ea39944f180f7b6509c0e5aa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c011e6a2271341dc9cf323fefacdd776" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44111 } } peers { permanent_uuid: "fd9ff8462e4446458f52aa36793a8d32" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46209 } } peers { permanent_uuid: "496e672ea39944f180f7b6509c0e5aa1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36347 } }
I20260620 02:57:13.293174 20226 raft_consensus.cc:374] T 4badb414158f4efa9e36bcbcc2c5965a P 496e672ea39944f180f7b6509c0e5aa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:13.293257 20226 raft_consensus.cc:729] T 4badb414158f4efa9e36bcbcc2c5965a P 496e672ea39944f180f7b6509c0e5aa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 496e672ea39944f180f7b6509c0e5aa1, State: Initialized, Role: FOLLOWER
I20260620 02:57:13.294857 20225 ts_tablet_manager.cc:1428] T 4badb414158f4efa9e36bcbcc2c5965a P fd9ff8462e4446458f52aa36793a8d32: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260620 02:57:13.295116 20226 consensus_queue.cc:260] T 4badb414158f4efa9e36bcbcc2c5965a P 496e672ea39944f180f7b6509c0e5aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c011e6a2271341dc9cf323fefacdd776" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44111 } } peers { permanent_uuid: "fd9ff8462e4446458f52aa36793a8d32" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46209 } } peers { permanent_uuid: "496e672ea39944f180f7b6509c0e5aa1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36347 } }
I20260620 02:57:13.296167 20073 heartbeater.cc:503] Master 127.0.97.62:44231 was elected leader, sending a full tablet report...
I20260620 02:57:13.296787 20226 ts_tablet_manager.cc:1428] T 4badb414158f4efa9e36bcbcc2c5965a P 496e672ea39944f180f7b6509c0e5aa1: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
W20260620 02:57:13.359256 19940 tablet.cc:2369] T 4badb414158f4efa9e36bcbcc2c5965a P fd9ff8462e4446458f52aa36793a8d32: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:57:13.376989 20074 tablet.cc:2369] T 4badb414158f4efa9e36bcbcc2c5965a P 496e672ea39944f180f7b6509c0e5aa1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:57:13.421952 20207 tablet.cc:2369] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:57:13.629304 20231 raft_consensus.cc:482] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:57:13.629634 20231 raft_consensus.cc:504] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c011e6a2271341dc9cf323fefacdd776" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44111 } } peers { permanent_uuid: "fd9ff8462e4446458f52aa36793a8d32" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46209 } } peers { permanent_uuid: "496e672ea39944f180f7b6509c0e5aa1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36347 } }
I20260620 02:57:13.631443 20231 leader_election.cc:288] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd9ff8462e4446458f52aa36793a8d32 (127.0.97.1:46209), 496e672ea39944f180f7b6509c0e5aa1 (127.0.97.2:36347)
I20260620 02:57:13.637563 20224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:13.632146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41331 (local address 127.0.97.2:36347)
0620 02:57:13.632356 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:13.632362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:13.632379 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:57:13.632448 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:13.632452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:13.632532 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:13.632630 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:57:13.633070 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:13.634051 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:13.635797 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:13.636686 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:13.636827 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:13.636922 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:13.637186 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 02:57:13.637310 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 02:57:13.637603 20209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:13.631745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36347 (local address 127.0.97.3:41331)
0620 02:57:13.632102 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:13.632122 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:13.632255 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:13.632710 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:13.632715 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:13.632743 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:57:13.632917 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:13.632926 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:13.634183 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:13.634191 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:13.635663 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:13.635675 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:13.636846 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:13.636852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:13.637028 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:13.637053 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 02:57:13.637161 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 02:57:13.637300 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238}
I20260620 02:57:13.638324 20222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:13.631925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50607 (local address 127.0.97.1:46209)
0620 02:57:13.632121 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:13.632127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:13.632606 (+ 479us) server_negotiation.cc:407] Connection header received
0620 02:57:13.632809 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:13.632814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:13.632872 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:13.632919 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:57:13.633682 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:13.634681 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:13.636497 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:13.637287 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:13.637455 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:13.637558 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:13.637970 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 02:57:13.638114 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 02:57:13.638360 20234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:13.631795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46209 (local address 127.0.97.3:50607)
0620 02:57:13.632479 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:13.632496 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:13.632637 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:13.633131 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:13.633142 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:13.633170 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:57:13.633518 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:13.633530 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:13.634876 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:13.634879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:13.636327 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:13.636339 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:13.637499 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:13.637506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:13.637734 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:13.637792 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 02:57:13.637912 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 02:57:13.638037 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":1207,"threads_started":1}
I20260620 02:57:13.638749 20028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4badb414158f4efa9e36bcbcc2c5965a" candidate_uuid: "c011e6a2271341dc9cf323fefacdd776" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "496e672ea39944f180f7b6509c0e5aa1" is_pre_election: true
I20260620 02:57:13.639335 20028 raft_consensus.cc:2454] T 4badb414158f4efa9e36bcbcc2c5965a P 496e672ea39944f180f7b6509c0e5aa1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c011e6a2271341dc9cf323fefacdd776 in term 0.
I20260620 02:57:13.639446 19894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4badb414158f4efa9e36bcbcc2c5965a" candidate_uuid: "c011e6a2271341dc9cf323fefacdd776" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd9ff8462e4446458f52aa36793a8d32" is_pre_election: true
I20260620 02:57:13.640020 19894 raft_consensus.cc:2454] T 4badb414158f4efa9e36bcbcc2c5965a P fd9ff8462e4446458f52aa36793a8d32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c011e6a2271341dc9cf323fefacdd776 in term 0.
I20260620 02:57:13.640036 20096 leader_election.cc:302] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 496e672ea39944f180f7b6509c0e5aa1, c011e6a2271341dc9cf323fefacdd776; no voters:
I20260620 02:57:13.640400 20231 raft_consensus.cc:2781] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:57:13.640556 20231 raft_consensus.cc:482] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:57:13.640673 20231 raft_consensus.cc:3037] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:13.642956 20231 raft_consensus.cc:504] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c011e6a2271341dc9cf323fefacdd776" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44111 } } peers { permanent_uuid: "fd9ff8462e4446458f52aa36793a8d32" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46209 } } peers { permanent_uuid: "496e672ea39944f180f7b6509c0e5aa1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36347 } }
I20260620 02:57:13.643604 20231 leader_election.cc:288] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776 [CANDIDATE]: Term 1 election: Requested vote from peers fd9ff8462e4446458f52aa36793a8d32 (127.0.97.1:46209), 496e672ea39944f180f7b6509c0e5aa1 (127.0.97.2:36347)
I20260620 02:57:13.644110 20028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4badb414158f4efa9e36bcbcc2c5965a" candidate_uuid: "c011e6a2271341dc9cf323fefacdd776" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "496e672ea39944f180f7b6509c0e5aa1"
I20260620 02:57:13.644104 19894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4badb414158f4efa9e36bcbcc2c5965a" candidate_uuid: "c011e6a2271341dc9cf323fefacdd776" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd9ff8462e4446458f52aa36793a8d32"
I20260620 02:57:13.644331 20028 raft_consensus.cc:3037] T 4badb414158f4efa9e36bcbcc2c5965a P 496e672ea39944f180f7b6509c0e5aa1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:13.644343 19894 raft_consensus.cc:3037] T 4badb414158f4efa9e36bcbcc2c5965a P fd9ff8462e4446458f52aa36793a8d32 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:13.647624 19894 raft_consensus.cc:2454] T 4badb414158f4efa9e36bcbcc2c5965a P fd9ff8462e4446458f52aa36793a8d32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c011e6a2271341dc9cf323fefacdd776 in term 1.
I20260620 02:57:13.647624 20028 raft_consensus.cc:2454] T 4badb414158f4efa9e36bcbcc2c5965a P 496e672ea39944f180f7b6509c0e5aa1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c011e6a2271341dc9cf323fefacdd776 in term 1.
I20260620 02:57:13.648077 20095 leader_election.cc:302] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c011e6a2271341dc9cf323fefacdd776, fd9ff8462e4446458f52aa36793a8d32; no voters:
I20260620 02:57:13.648398 20231 raft_consensus.cc:2781] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:57:13.648821 20231 raft_consensus.cc:686] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776 [term 1 LEADER]: Becoming Leader. State: Replica: c011e6a2271341dc9cf323fefacdd776, State: Running, Role: LEADER
I20260620 02:57:13.649367 20231 consensus_queue.cc:237] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c011e6a2271341dc9cf323fefacdd776" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44111 } } peers { permanent_uuid: "fd9ff8462e4446458f52aa36793a8d32" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46209 } } peers { permanent_uuid: "496e672ea39944f180f7b6509c0e5aa1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36347 } }
I20260620 02:57:13.653846 19748 catalog_manager.cc:5495] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776 reported cstate change: term changed from 0 to 1, leader changed from <none> to c011e6a2271341dc9cf323fefacdd776 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "c011e6a2271341dc9cf323fefacdd776" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c011e6a2271341dc9cf323fefacdd776" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd9ff8462e4446458f52aa36793a8d32" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "496e672ea39944f180f7b6509c0e5aa1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36347 } health_report { overall_health: UNKNOWN } } }
I20260620 02:57:13.710477 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:13.704830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32928 (local address 127.0.97.3:44111)
0620 02:57:13.705051 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:13.705055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:13.705072 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:57:13.705179 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:13.705182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:13.705257 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:13.705383 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:57:13.705849 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:13.706697 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:13.708207 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:13.708621 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:13.708722 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:13.709281 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:57:13.709825 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:57:13.709959 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:13.710132 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 02:57:13.710204 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 02:57:13.719532 19894 raft_consensus.cc:1261] T 4badb414158f4efa9e36bcbcc2c5965a P fd9ff8462e4446458f52aa36793a8d32 [term 1 FOLLOWER]: Refusing update from remote peer c011e6a2271341dc9cf323fefacdd776: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:57:13.719534 20028 raft_consensus.cc:1261] T 4badb414158f4efa9e36bcbcc2c5965a P 496e672ea39944f180f7b6509c0e5aa1 [term 1 FOLLOWER]: Refusing update from remote peer c011e6a2271341dc9cf323fefacdd776: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:57:13.720598 20235 consensus_queue.cc:1035] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776 [LEADER]: Connected to new peer: Peer: permanent_uuid: "496e672ea39944f180f7b6509c0e5aa1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:57:13.720937 20231 consensus_queue.cc:1035] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd9ff8462e4446458f52aa36793a8d32" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:57:13.745927 19750 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:57:13.746243 19750 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:57:13.749771 19750 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd7f01ccf152411aba47dfa0bd884c3f: Sending DeleteTablet for 3 replicas of tablet 4badb414158f4efa9e36bcbcc2c5965a
I20260620 02:57:13.751052 19874 tablet_service.cc:1514] Processing DeleteTablet for tablet 4badb414158f4efa9e36bcbcc2c5965a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:57:13 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52880
I20260620 02:57:13.751178 20141 tablet_service.cc:1514] Processing DeleteTablet for tablet 4badb414158f4efa9e36bcbcc2c5965a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:57:13 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:32918
I20260620 02:57:13.752403 20008 tablet_service.cc:1514] Processing DeleteTablet for tablet 4badb414158f4efa9e36bcbcc2c5965a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:57:13 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41952
I20260620 02:57:13.753252 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 19811
I20260620 02:57:13.763844 20248 tablet_replica.cc:330] T 4badb414158f4efa9e36bcbcc2c5965a P fd9ff8462e4446458f52aa36793a8d32: stopping tablet replica
I20260620 02:57:13.764446 20248 raft_consensus.cc:2229] T 4badb414158f4efa9e36bcbcc2c5965a P fd9ff8462e4446458f52aa36793a8d32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:57:13.765050 20248 raft_consensus.cc:2258] T 4badb414158f4efa9e36bcbcc2c5965a P fd9ff8462e4446458f52aa36793a8d32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:57:13.765962 20249 tablet_replica.cc:330] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776: stopping tablet replica
I20260620 02:57:13.766486 20249 raft_consensus.cc:2229] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776 [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:57:13.767096 20250 tablet_replica.cc:330] T 4badb414158f4efa9e36bcbcc2c5965a P 496e672ea39944f180f7b6509c0e5aa1: stopping tablet replica
I20260620 02:57:13.767313 20249 raft_consensus.cc:2258] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:57:13.767495 20250 raft_consensus.cc:2229] T 4badb414158f4efa9e36bcbcc2c5965a P 496e672ea39944f180f7b6509c0e5aa1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:57:13.767992 20250 raft_consensus.cc:2258] T 4badb414158f4efa9e36bcbcc2c5965a P 496e672ea39944f180f7b6509c0e5aa1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:57:13.769905 20249 ts_tablet_manager.cc:1905] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:57:13.770015 20250 ts_tablet_manager.cc:1905] T 4badb414158f4efa9e36bcbcc2c5965a P 496e672ea39944f180f7b6509c0e5aa1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:57:13.775910 20248 ts_tablet_manager.cc:1905] T 4badb414158f4efa9e36bcbcc2c5965a P fd9ff8462e4446458f52aa36793a8d32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:57:13.777343 20249 ts_tablet_manager.cc:1918] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:57:13.777575 20249 log.cc:1196] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-2/wal/wals/4badb414158f4efa9e36bcbcc2c5965a
I20260620 02:57:13.778020 20250 ts_tablet_manager.cc:1918] T 4badb414158f4efa9e36bcbcc2c5965a P 496e672ea39944f180f7b6509c0e5aa1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:57:13.778234 20250 log.cc:1196] T 4badb414158f4efa9e36bcbcc2c5965a P 496e672ea39944f180f7b6509c0e5aa1: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-1/wal/wals/4badb414158f4efa9e36bcbcc2c5965a
I20260620 02:57:13.778249 20249 ts_tablet_manager.cc:1939] T 4badb414158f4efa9e36bcbcc2c5965a P c011e6a2271341dc9cf323fefacdd776: Deleting consensus metadata
I20260620 02:57:13.778860 20250 ts_tablet_manager.cc:1939] T 4badb414158f4efa9e36bcbcc2c5965a P 496e672ea39944f180f7b6509c0e5aa1: Deleting consensus metadata
I20260620 02:57:13.780282 19736 catalog_manager.cc:4841] TS c011e6a2271341dc9cf323fefacdd776 (127.0.97.3:44111): tablet 4badb414158f4efa9e36bcbcc2c5965a (table test-table [id=bc15692b71c645a386f498b4094fd18e]) successfully deleted
I20260620 02:57:13.780659 19736 catalog_manager.cc:4841] TS 496e672ea39944f180f7b6509c0e5aa1 (127.0.97.2:36347): tablet 4badb414158f4efa9e36bcbcc2c5965a (table test-table [id=bc15692b71c645a386f498b4094fd18e]) successfully deleted
I20260620 02:57:13.783315 20248 ts_tablet_manager.cc:1918] T 4badb414158f4efa9e36bcbcc2c5965a P fd9ff8462e4446458f52aa36793a8d32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:57:13.783500 20248 log.cc:1196] T 4badb414158f4efa9e36bcbcc2c5965a P fd9ff8462e4446458f52aa36793a8d32: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781923914418765-388-0/minicluster-data/ts-0/wal/wals/4badb414158f4efa9e36bcbcc2c5965a
I20260620 02:57:13.783937 20248 ts_tablet_manager.cc:1939] T 4badb414158f4efa9e36bcbcc2c5965a P fd9ff8462e4446458f52aa36793a8d32: Deleting consensus metadata
Jun 20 02:57:13 dist-test-slave-8p8c krb5kdc[19705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 02:57:13.785475 19735 catalog_manager.cc:4841] TS fd9ff8462e4446458f52aa36793a8d32 (127.0.97.1:46209): tablet 4badb414158f4efa9e36bcbcc2c5965a (table test-table [id=bc15692b71c645a386f498b4094fd18e]) successfully deleted
I20260620 02:57:13.791713 20222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:13.754994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52888 (local address 127.0.97.1:46209)
0620 02:57:13.755207 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:13.755217 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:13.766383 (+ 11166us) server_negotiation.cc:407] Connection header received
0620 02:57:13.766614 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:13.766622 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:13.766733 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:13.766856 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:13.769057 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:13.770153 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:13.772105 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:13.772752 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:13.772848 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:13.785407 (+ 12559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:13.785420 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:13.785438 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:13.785493 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:13.788465 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:13.789476 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:13.789485 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:13.789487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:13.789639 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:13.790108 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:13.790112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:13.790115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:13.790600 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:13.790838 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:13.791221 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 02:57:13.791389 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 02:57:13.792572 19934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:57:13.885262 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 19811
I20260620 02:57:13.903975 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 19945
Jun 20 02:57:13 dist-test-slave-8p8c krb5kdc[19705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 02:57:13.922921 20224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:13.904870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41964 (local address 127.0.97.2:36347)
0620 02:57:13.905084 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:13.905090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:13.905114 (+ 24us) server_negotiation.cc:407] Connection header received
0620 02:57:13.905239 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:13.905246 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:13.905339 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:13.905467 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:13.906866 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:13.908083 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:13.909392 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:13.910106 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:13.910251 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:13.915291 (+ 5040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:13.915313 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:13.915336 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:13.915386 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:13.919587 (+ 4201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:13.920445 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:13.920456 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:13.920460 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:13.920662 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:13.921033 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:13.921039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:13.921043 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:13.921629 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:13.921970 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:13.922211 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 02:57:13.922489 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 02:57:13.923669 20068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:57:14.015782 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 19945
I20260620 02:57:14.034639 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 20078
Jun 20 02:57:14 dist-test-slave-8p8c krb5kdc[19705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 02:57:14.054883 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:14.035516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32940 (local address 127.0.97.3:44111)
0620 02:57:14.035690 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:14.035698 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:14.035715 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:57:14.035930 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:14.035935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:14.036015 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:14.036157 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:14.037929 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:14.038904 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:14.040701 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:14.041193 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:14.041293 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:14.047220 (+ 5927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:14.047236 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:14.047258 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:14.047325 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:14.051507 (+ 4182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:14.052241 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:14.052252 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:14.052256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:14.052465 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:14.052836 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:14.052841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:14.052845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:14.053311 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:14.053646 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:14.053956 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 02:57:14.054180 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 02:57:14.055733 20201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:57:14.179162 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 20078
I20260620 02:57:14.199432 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 19718
I20260620 02:57:14.200683 19780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:57:14.313483 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 19718
2026-06-20T02:57:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4666 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T02:57:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:57:14Z Disabled control of system clock
I20260620 02:57:14.391607 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39399
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:46825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 02:57:14.730824 20259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:14.731240 20259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:14.731437 20259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:14.740149 20259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:57:14.740267 20259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:14.740329 20259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:14.740406 20259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:57:14.740463 20259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:57:14.750191 20259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39399
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:14.751585 20259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:14.753643 20259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:14.766395 20267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:14.766449 20265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:14.766709 20264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:14.767478 20259 server_base.cc:1034] running on GCE node
I20260620 02:57:14.768339 20259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:14.769829 20259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:14.771055 20259 hybrid_clock.cc:648] HybridClock initialized: now 1781924234771009 us; error 43 us; skew 500 ppm
I20260620 02:57:14.771454 20259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:57:14.774328 20259 webserver.cc:460] Webserver started at http://127.0.97.62:34249/ using document root <none> and password file <none>
I20260620 02:57:14.775182 20259 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:14.775331 20259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:14.775695 20259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:14.779387 20259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "f4fbf36b122643b6b60a0ee476dbc278"
format_stamp: "Formatted at 2026-06-20 02:57:14 on dist-test-slave-8p8c"
server_key: "992d08d2487ef5e9526362f80a20c185"
server_key_iv: "5b5106644f9622d23a35c8ed270aaad0"
server_key_version: "encryptionkey@0"
I20260620 02:57:14.780324 20259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "f4fbf36b122643b6b60a0ee476dbc278"
format_stamp: "Formatted at 2026-06-20 02:57:14 on dist-test-slave-8p8c"
server_key: "992d08d2487ef5e9526362f80a20c185"
server_key_iv: "5b5106644f9622d23a35c8ed270aaad0"
server_key_version: "encryptionkey@0"
I20260620 02:57:14.786783 20259 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260620 02:57:14.790627 20273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:14.792354 20259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:57:14.792608 20259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "f4fbf36b122643b6b60a0ee476dbc278"
format_stamp: "Formatted at 2026-06-20 02:57:14 on dist-test-slave-8p8c"
server_key: "992d08d2487ef5e9526362f80a20c185"
server_key_iv: "5b5106644f9622d23a35c8ed270aaad0"
server_key_version: "encryptionkey@0"
I20260620 02:57:14.792850 20259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:14.806842 20259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:14.807740 20259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:14.808034 20259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:14.829263 20259 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39399
I20260620 02:57:14.829299 20324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39399 every 8 connection(s)
I20260620 02:57:14.830760 20259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:57:14.835002 20325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:14.835526 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 20259
I20260620 02:57:14.835716 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:57:14.836298 388 external_mini_cluster.cc:1428] Setting key b30722f86254dfc3784948d2200aebaf
I20260620 02:57:14.848874 20325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4fbf36b122643b6b60a0ee476dbc278: Bootstrap starting.
I20260620 02:57:14.851424 20328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:14.838540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34318 (local address 127.0.97.62:39399)
0620 02:57:14.839679 (+ 1139us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:14.839690 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:14.839738 (+ 48us) server_negotiation.cc:407] Connection header received
0620 02:57:14.842104 (+ 2366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:14.842131 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:14.842871 (+ 740us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:14.843358 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:14.843946 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:14.845208 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:14.846620 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:14.847583 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:14.847811 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:14.849043 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:14.849053 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:14.849076 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:14.849629 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:14.849707 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:14.849727 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:14.849848 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:14.850045 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 02:57:14.850425 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":421,"threads_started":1}
I20260620 02:57:14.852357 20325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4fbf36b122643b6b60a0ee476dbc278: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:14.853535 20325 log.cc:824] T 00000000000000000000000000000000 P f4fbf36b122643b6b60a0ee476dbc278: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:14.856412 20325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4fbf36b122643b6b60a0ee476dbc278: No bootstrap required, opened a new log
I20260620 02:57:14.863991 20325 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4fbf36b122643b6b60a0ee476dbc278 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4fbf36b122643b6b60a0ee476dbc278" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39399 } }
I20260620 02:57:14.864439 20325 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4fbf36b122643b6b60a0ee476dbc278 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:14.864560 20325 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4fbf36b122643b6b60a0ee476dbc278 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4fbf36b122643b6b60a0ee476dbc278, State: Initialized, Role: FOLLOWER
I20260620 02:57:14.865155 20325 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4fbf36b122643b6b60a0ee476dbc278 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4fbf36b122643b6b60a0ee476dbc278" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39399 } }
I20260620 02:57:14.865408 20325 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4fbf36b122643b6b60a0ee476dbc278 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:57:14.865554 20325 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4fbf36b122643b6b60a0ee476dbc278 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:57:14.865736 20325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4fbf36b122643b6b60a0ee476dbc278 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:14.868059 20325 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4fbf36b122643b6b60a0ee476dbc278 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4fbf36b122643b6b60a0ee476dbc278" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39399 } }
I20260620 02:57:14.868595 20325 leader_election.cc:302] T 00000000000000000000000000000000 P f4fbf36b122643b6b60a0ee476dbc278 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4fbf36b122643b6b60a0ee476dbc278; no voters:
I20260620 02:57:14.869309 20325 leader_election.cc:288] T 00000000000000000000000000000000 P f4fbf36b122643b6b60a0ee476dbc278 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:57:14.869427 20330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4fbf36b122643b6b60a0ee476dbc278 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:57:14.870139 20330 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4fbf36b122643b6b60a0ee476dbc278 [term 1 LEADER]: Becoming Leader. State: Replica: f4fbf36b122643b6b60a0ee476dbc278, State: Running, Role: LEADER
I20260620 02:57:14.870787 20330 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4fbf36b122643b6b60a0ee476dbc278 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4fbf36b122643b6b60a0ee476dbc278" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39399 } }
I20260620 02:57:14.871796 20325 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4fbf36b122643b6b60a0ee476dbc278 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:57:14.873785 20332 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4fbf36b122643b6b60a0ee476dbc278 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4fbf36b122643b6b60a0ee476dbc278. Latest consensus state: current_term: 1 leader_uuid: "f4fbf36b122643b6b60a0ee476dbc278" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4fbf36b122643b6b60a0ee476dbc278" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39399 } } }
I20260620 02:57:14.874205 20332 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4fbf36b122643b6b60a0ee476dbc278 [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:14.874020 20331 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4fbf36b122643b6b60a0ee476dbc278 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4fbf36b122643b6b60a0ee476dbc278" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4fbf36b122643b6b60a0ee476dbc278" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39399 } } }
I20260620 02:57:14.874406 20331 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4fbf36b122643b6b60a0ee476dbc278 [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:14.876821 20337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:57:14.882304 20337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:57:14.892000 20337 catalog_manager.cc:1347] Generated new cluster ID: a34562410e6f4c09a7faff2c33cfb8f6
I20260620 02:57:14.892136 20337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:57:14.904135 20337 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:57:14.905470 20337 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:57:14.915660 20337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4fbf36b122643b6b60a0ee476dbc278: Generated new TSK 0
I20260620 02:57:14.916564 20337 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:57:14.924891 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39399
--builtin_ntp_servers=127.0.97.20:46825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 02:57:15.269222 20349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:15.269527 20349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:15.269695 20349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:15.277452 20349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:15.277572 20349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:15.277735 20349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 02:57:15.287426 20349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:15.288803 20349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:15.290896 20349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:15.304319 20354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:15.304435 20355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:15.304869 20357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:15.305110 20349 server_base.cc:1034] running on GCE node
I20260620 02:57:15.305752 20349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:15.306761 20349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:15.308056 20349 hybrid_clock.cc:648] HybridClock initialized: now 1781924235308004 us; error 56 us; skew 500 ppm
I20260620 02:57:15.308384 20349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:57:15.310899 20349 webserver.cc:460] Webserver started at http://127.0.97.1:45209/ using document root <none> and password file <none>
I20260620 02:57:15.311606 20349 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:15.311774 20349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:15.312153 20349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:15.315183 20349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-0/data/instance:
uuid: "d1cd48847bdf47bd97f0389d5352967b"
format_stamp: "Formatted at 2026-06-20 02:57:15 on dist-test-slave-8p8c"
server_key: "3414322526cfe5e671bfbab6bbdf6744"
server_key_iv: "49f002cfb443816b0af7668e9313c66f"
server_key_version: "encryptionkey@0"
I20260620 02:57:15.316005 20349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d1cd48847bdf47bd97f0389d5352967b"
format_stamp: "Formatted at 2026-06-20 02:57:15 on dist-test-slave-8p8c"
server_key: "3414322526cfe5e671bfbab6bbdf6744"
server_key_iv: "49f002cfb443816b0af7668e9313c66f"
server_key_version: "encryptionkey@0"
I20260620 02:57:15.321368 20349 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:57:15.324985 20363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:15.326349 20349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:57:15.326598 20349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-0/wal
uuid: "d1cd48847bdf47bd97f0389d5352967b"
format_stamp: "Formatted at 2026-06-20 02:57:15 on dist-test-slave-8p8c"
server_key: "3414322526cfe5e671bfbab6bbdf6744"
server_key_iv: "49f002cfb443816b0af7668e9313c66f"
server_key_version: "encryptionkey@0"
I20260620 02:57:15.326828 20349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:15.337708 20349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:15.338600 20349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:15.338876 20349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:15.339905 20349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:57:15.341387 20349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:57:15.341504 20349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:15.341637 20349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:57:15.341722 20349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:15.377018 20349 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43923
I20260620 02:57:15.377081 20475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43923 every 8 connection(s)
I20260620 02:57:15.378505 20349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-0/data/info.pb
I20260620 02:57:15.383782 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 20349
I20260620 02:57:15.383946 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-0/wal/instance
I20260620 02:57:15.384473 388 external_mini_cluster.cc:1428] Setting key 1e3e180f0ce5cfcc5b95909c91f54d6e
I20260620 02:57:15.387655 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39399
--builtin_ntp_servers=127.0.97.20:46825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 02:57:15.392956 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:15.381114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59179 (local address 127.0.97.62:39399)
0620 02:57:15.381547 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:15.381551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:15.382763 (+ 1212us) server_negotiation.cc:407] Connection header received
0620 02:57:15.385153 (+ 2390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:15.385159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:15.385267 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:15.385393 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:15.386865 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:15.387967 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:15.389515 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:15.390219 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:15.390358 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:15.391933 (+ 1575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:15.391942 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:15.391950 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:15.391977 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:15.392029 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:15.392047 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:15.392139 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:15.392375 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 02:57:15.392489 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":237,"threads_started":1}
I20260620 02:57:15.393829 20479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:15.381662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39399 (local address 127.0.97.1:59179)
0620 02:57:15.382614 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:15.382651 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:15.384840 (+ 2189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:15.385691 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:15.385702 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:15.385826 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:57:15.386697 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:15.386719 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:15.388122 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:15.388126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:15.389365 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:15.389378 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:15.390524 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:15.390531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:15.390792 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:15.391584 (+ 792us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:57:15.391587 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:57:15.391641 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:57:15.391724 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:57:15.392159 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:57:15.392168 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:57:15.392174 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 02:57:15.392320 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 02:57:15.392631 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":272,"threads_started":1}
I20260620 02:57:15.395444 20476 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39399
I20260620 02:57:15.395792 20476 heartbeater.cc:464] Registering TS with master...
I20260620 02:57:15.396761 20476 heartbeater.cc:511] Master 127.0.97.62:39399 requested a full tablet report, sending...
I20260620 02:57:15.399008 20290 ts_manager.cc:194] Registered new tserver with Master: d1cd48847bdf47bd97f0389d5352967b (127.0.97.1:43923)
I20260620 02:57:15.400511 20290 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:59179
W20260620 02:57:15.721206 20480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:15.721532 20480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:15.721688 20480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:15.729080 20480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:15.729219 20480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:15.729386 20480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 02:57:15.738683 20480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:15.740036 20480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:15.741803 20480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:15.755138 20485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:15.755190 20486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:15.755216 20488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:15.756294 20480 server_base.cc:1034] running on GCE node
I20260620 02:57:15.756953 20480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:15.757990 20480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:15.759255 20480 hybrid_clock.cc:648] HybridClock initialized: now 1781924235759201 us; error 45 us; skew 500 ppm
I20260620 02:57:15.759642 20480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:57:15.762140 20480 webserver.cc:460] Webserver started at http://127.0.97.2:34773/ using document root <none> and password file <none>
I20260620 02:57:15.762948 20480 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:15.763083 20480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:15.763415 20480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:15.766722 20480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-1/data/instance:
uuid: "004e9d427aac45f7a51ded8b76cea407"
format_stamp: "Formatted at 2026-06-20 02:57:15 on dist-test-slave-8p8c"
server_key: "11f561878ffdc63bd60c3aa9d05feaad"
server_key_iv: "f9aea236bbf1b53c2c8b658bcbef3ed4"
server_key_version: "encryptionkey@0"
I20260620 02:57:15.767585 20480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-1/wal/instance:
uuid: "004e9d427aac45f7a51ded8b76cea407"
format_stamp: "Formatted at 2026-06-20 02:57:15 on dist-test-slave-8p8c"
server_key: "11f561878ffdc63bd60c3aa9d05feaad"
server_key_iv: "f9aea236bbf1b53c2c8b658bcbef3ed4"
server_key_version: "encryptionkey@0"
I20260620 02:57:15.773226 20480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:57:15.777264 20494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:15.778949 20480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 02:57:15.779161 20480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-1/wal
uuid: "004e9d427aac45f7a51ded8b76cea407"
format_stamp: "Formatted at 2026-06-20 02:57:15 on dist-test-slave-8p8c"
server_key: "11f561878ffdc63bd60c3aa9d05feaad"
server_key_iv: "f9aea236bbf1b53c2c8b658bcbef3ed4"
server_key_version: "encryptionkey@0"
I20260620 02:57:15.779403 20480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:15.798950 20480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:15.799877 20480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:15.800220 20480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:15.801218 20480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:57:15.802909 20480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:57:15.803035 20480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:15.803161 20480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:57:15.803264 20480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:15.843050 20480 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37575
I20260620 02:57:15.843140 20606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37575 every 8 connection(s)
I20260620 02:57:15.844412 20480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-1/data/info.pb
I20260620 02:57:15.845821 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 20480
I20260620 02:57:15.845997 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-1/wal/instance
I20260620 02:57:15.846553 388 external_mini_cluster.cc:1428] Setting key 3bdf4bada5d7ec11fc261083fa75c087
I20260620 02:57:15.849530 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39399
--builtin_ntp_servers=127.0.97.20:46825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 02:57:15.859333 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:15.848489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43983 (local address 127.0.97.62:39399)
0620 02:57:15.848705 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:15.848712 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:15.849963 (+ 1251us) server_negotiation.cc:407] Connection header received
0620 02:57:15.851869 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:15.851877 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:15.851969 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:15.852104 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:15.853247 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:15.854498 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:15.856220 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:15.856624 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:15.856764 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:15.858557 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:15.858562 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:15.858565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:15.858585 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:15.858618 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:15.858630 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:15.858714 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:15.858974 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 02:57:15.859063 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 02:57:15.860868 20609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:15.848707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39399 (local address 127.0.97.2:43983)
0620 02:57:15.849798 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:15.849845 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:15.851571 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:15.852350 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:15.852358 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:15.852456 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:57:15.853066 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:15.853086 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:15.854686 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:15.854691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:15.856036 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:15.856051 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:15.856860 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:15.856867 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:15.857118 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:15.857976 (+ 858us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:57:15.857981 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:57:15.858045 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:57:15.858276 (+ 231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:57:15.858740 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:57:15.858753 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:57:15.858764 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 02:57:15.858963 (+ 199us) client_negotiation.cc:240] Negotiation successful
0620 02:57:15.859430 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":339,"threads_started":1}
I20260620 02:57:15.862548 20607 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39399
I20260620 02:57:15.862886 20607 heartbeater.cc:464] Registering TS with master...
I20260620 02:57:15.863713 20607 heartbeater.cc:511] Master 127.0.97.62:39399 requested a full tablet report, sending...
I20260620 02:57:15.865223 20290 ts_manager.cc:194] Registered new tserver with Master: 004e9d427aac45f7a51ded8b76cea407 (127.0.97.2:37575)
I20260620 02:57:15.866057 20290 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:43983
W20260620 02:57:16.190608 20610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:16.190953 20610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:16.191116 20610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:16.198535 20610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:16.198670 20610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:16.198845 20610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 02:57:16.208071 20610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:16.209481 20610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:16.211563 20610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:16.225652 20618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:16.225726 20615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:16.225762 20616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:16.226244 20610 server_base.cc:1034] running on GCE node
I20260620 02:57:16.226980 20610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:16.227967 20610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:16.229252 20610 hybrid_clock.cc:648] HybridClock initialized: now 1781924236229176 us; error 70 us; skew 500 ppm
I20260620 02:57:16.229629 20610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:57:16.232250 20610 webserver.cc:460] Webserver started at http://127.0.97.3:34067/ using document root <none> and password file <none>
I20260620 02:57:16.232937 20610 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:16.233039 20610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:16.233302 20610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:16.236276 20610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-2/data/instance:
uuid: "0b669241035c42f1b0a1ea7be21bf80d"
format_stamp: "Formatted at 2026-06-20 02:57:16 on dist-test-slave-8p8c"
server_key: "db364777cca3059b5ab61f5901dc90dd"
server_key_iv: "6744884e264932014db622e181dc7b15"
server_key_version: "encryptionkey@0"
I20260620 02:57:16.237030 20610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-2/wal/instance:
uuid: "0b669241035c42f1b0a1ea7be21bf80d"
format_stamp: "Formatted at 2026-06-20 02:57:16 on dist-test-slave-8p8c"
server_key: "db364777cca3059b5ab61f5901dc90dd"
server_key_iv: "6744884e264932014db622e181dc7b15"
server_key_version: "encryptionkey@0"
I20260620 02:57:16.242400 20610 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 02:57:16.246296 20624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:16.248154 20610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 02:57:16.248368 20610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-2/wal
uuid: "0b669241035c42f1b0a1ea7be21bf80d"
format_stamp: "Formatted at 2026-06-20 02:57:16 on dist-test-slave-8p8c"
server_key: "db364777cca3059b5ab61f5901dc90dd"
server_key_iv: "6744884e264932014db622e181dc7b15"
server_key_version: "encryptionkey@0"
I20260620 02:57:16.248639 20610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:16.283392 20610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:16.284374 20610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:16.284669 20610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:16.285745 20610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 02:57:16.287442 20610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 02:57:16.287537 20610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:16.287704 20610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 02:57:16.287791 20610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:16.326995 20610 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46397
I20260620 02:57:16.327097 20736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46397 every 8 connection(s)
I20260620 02:57:16.328399 20610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-2/data/info.pb
I20260620 02:57:16.338523 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 20610
I20260620 02:57:16.338735 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-2/wal/instance
I20260620 02:57:16.339165 388 external_mini_cluster.cc:1428] Setting key f11c6d5de6892fb1709c35732bf6baf7
I20260620 02:57:16.341931 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:16.331232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46361 (local address 127.0.97.62:39399)
0620 02:57:16.331454 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:16.331459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:16.332921 (+ 1462us) server_negotiation.cc:407] Connection header received
0620 02:57:16.334801 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:16.334807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:16.334897 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:16.335008 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:16.336089 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.336827 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.338635 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.339186 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.339336 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:16.341039 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:16.341046 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:16.341050 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:16.341076 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:16.341111 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:16.341127 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:16.341236 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:16.341463 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 02:57:16.341563 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 02:57:16.343075 20739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:16.331810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39399 (local address 127.0.97.3:46361)
0620 02:57:16.332764 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:16.332799 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:16.334523 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:16.335251 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:16.335259 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:16.335352 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:57:16.335920 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:16.335935 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.336978 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.336983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:16.338440 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:16.338457 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.339425 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.339433 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:16.339714 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:16.340702 (+ 988us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 02:57:16.340705 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:57:16.340749 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 02:57:16.340831 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:57:16.341233 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:57:16.341246 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:57:16.341257 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 02:57:16.341444 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 02:57:16.341876 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":329,"threads_started":1}
I20260620 02:57:16.344614 20737 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39399
I20260620 02:57:16.344959 20737 heartbeater.cc:464] Registering TS with master...
I20260620 02:57:16.345803 20737 heartbeater.cc:511] Master 127.0.97.62:39399 requested a full tablet report, sending...
I20260620 02:57:16.347373 20290 ts_manager.cc:194] Registered new tserver with Master: 0b669241035c42f1b0a1ea7be21bf80d (127.0.97.3:46397)
I20260620 02:57:16.348174 20290 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:46361
I20260620 02:57:16.354440 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 02:57:16.365469 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:16.359141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34326 (local address 127.0.97.62:39399)
0620 02:57:16.359362 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:16.359367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:16.359858 (+ 491us) server_negotiation.cc:407] Connection header received
0620 02:57:16.360040 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:16.360044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:16.360131 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:16.360216 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:16.360746 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.361471 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.362983 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.363408 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.363510 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:16.364793 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:16.364798 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:16.364802 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:16.364821 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:16.364855 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:16.364868 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:16.364947 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:16.365127 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 02:57:16.365242 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:57:16.369382 20290 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 02:57:16.376608 20290 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 02:57:16.404242 20476 heartbeater.cc:503] Master 127.0.97.62:39399 was elected leader, sending a full tablet report...
I20260620 02:57:16.412585 20751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:16.390066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43923 (local address 127.0.0.1:47274)
0620 02:57:16.401106 (+ 11040us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:16.401167 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:16.401368 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:16.405470 (+ 4102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:16.405478 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:16.405542 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:57:16.405998 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:16.406011 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.408227 (+ 2216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.408232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:16.410239 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:16.410251 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.411639 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.411648 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:16.411871 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:16.411929 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 02:57:16.412043 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 02:57:16.412233 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10764,"thread_start_us":15001,"threads_started":1}
I20260620 02:57:16.414204 20750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:16.390262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47274 (local address 127.0.97.1:43923)
0620 02:57:16.403710 (+ 13448us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:16.403723 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:16.403769 (+ 46us) server_negotiation.cc:407] Connection header received
0620 02:57:16.403925 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:16.403937 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:16.405203 (+ 1266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:16.406208 (+ 1005us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:57:16.406481 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.408075 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.410496 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.411403 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.413303 (+ 1900us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:16.413593 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:16.413665 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 02:57:16.413842 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13170,"thread_start_us":12930,"threads_started":1}
I20260620 02:57:16.414201 20753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:16.390071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37575 (local address 127.0.0.1:58796)
0620 02:57:16.404590 (+ 14519us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:16.404616 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:16.404810 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:16.407001 (+ 2191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:16.407005 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:16.407034 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:57:16.407531 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:16.407544 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.409438 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.409443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:16.411600 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:16.411610 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.413346 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.413352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:16.413527 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:16.413570 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 02:57:16.413701 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 02:57:16.413828 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14310,"mutex_wait_us":174,"thread_start_us":10424,"threads_started":1}
I20260620 02:57:16.414279 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:16.390441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58796 (local address 127.0.97.2:37575)
0620 02:57:16.403002 (+ 12561us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:16.403014 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:16.406449 (+ 3435us) server_negotiation.cc:407] Connection header received
0620 02:57:16.406568 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:16.406578 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:16.406817 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:16.407009 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:57:16.407716 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.409284 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.411788 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.413148 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.413295 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:16.413542 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:16.413745 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 02:57:16.413903 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12391,"thread_start_us":12288,"threads_started":1}
I20260620 02:57:16.414898 20754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:16.392157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55086 (local address 127.0.97.3:46397)
0620 02:57:16.403975 (+ 11818us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:16.403988 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:16.405742 (+ 1754us) server_negotiation.cc:407] Connection header received
0620 02:57:16.405905 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:16.405914 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:16.406206 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:16.406426 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:57:16.407407 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.408854 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.411586 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.412991 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.413135 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:16.413434 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:16.414309 (+ 875us) server_negotiation.cc:299] Negotiation successful
0620 02:57:16.414505 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11606,"thread_start_us":11447,"threads_started":1}
I20260620 02:57:16.415359 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:16.391703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46397 (local address 127.0.0.1:55086)
0620 02:57:16.405581 (+ 13878us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:16.405598 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:16.405751 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:16.406719 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:16.406723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:16.406751 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:57:16.407151 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:16.407165 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.409037 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.409042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:16.411346 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:16.411360 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.413568 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.413576 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:16.413784 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:16.413816 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 02:57:16.413893 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 02:57:16.414036 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13697,"thread_start_us":12467,"threads_started":1}
I20260620 02:57:16.418602 20672 tablet_service.cc:1467] Processing CreateTablet for tablet d6253f05a17641889fdd0295a9e1090c (DEFAULT_TABLE table=test-table [id=d6ac77be8306411e924839496ab626b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:57:16.418726 20542 tablet_service.cc:1467] Processing CreateTablet for tablet d6253f05a17641889fdd0295a9e1090c (DEFAULT_TABLE table=test-table [id=d6ac77be8306411e924839496ab626b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:57:16.418726 20411 tablet_service.cc:1467] Processing CreateTablet for tablet d6253f05a17641889fdd0295a9e1090c (DEFAULT_TABLE table=test-table [id=d6ac77be8306411e924839496ab626b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 02:57:16.420194 20672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6253f05a17641889fdd0295a9e1090c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:16.420210 20411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6253f05a17641889fdd0295a9e1090c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:16.420212 20542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6253f05a17641889fdd0295a9e1090c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:16.439168 20755 tablet_bootstrap.cc:492] T d6253f05a17641889fdd0295a9e1090c P 0b669241035c42f1b0a1ea7be21bf80d: Bootstrap starting.
I20260620 02:57:16.442621 20755 tablet_bootstrap.cc:654] T d6253f05a17641889fdd0295a9e1090c P 0b669241035c42f1b0a1ea7be21bf80d: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:16.443176 20756 tablet_bootstrap.cc:492] T d6253f05a17641889fdd0295a9e1090c P 004e9d427aac45f7a51ded8b76cea407: Bootstrap starting.
I20260620 02:57:16.445189 20757 tablet_bootstrap.cc:492] T d6253f05a17641889fdd0295a9e1090c P d1cd48847bdf47bd97f0389d5352967b: Bootstrap starting.
I20260620 02:57:16.444231 20755 log.cc:824] T d6253f05a17641889fdd0295a9e1090c P 0b669241035c42f1b0a1ea7be21bf80d: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:16.446799 20756 tablet_bootstrap.cc:654] T d6253f05a17641889fdd0295a9e1090c P 004e9d427aac45f7a51ded8b76cea407: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:16.447959 20756 log.cc:824] T d6253f05a17641889fdd0295a9e1090c P 004e9d427aac45f7a51ded8b76cea407: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:16.448954 20757 tablet_bootstrap.cc:654] T d6253f05a17641889fdd0295a9e1090c P d1cd48847bdf47bd97f0389d5352967b: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:16.449072 20755 tablet_bootstrap.cc:492] T d6253f05a17641889fdd0295a9e1090c P 0b669241035c42f1b0a1ea7be21bf80d: No bootstrap required, opened a new log
I20260620 02:57:16.449380 20755 ts_tablet_manager.cc:1397] T d6253f05a17641889fdd0295a9e1090c P 0b669241035c42f1b0a1ea7be21bf80d: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260620 02:57:16.450023 20757 log.cc:824] T d6253f05a17641889fdd0295a9e1090c P d1cd48847bdf47bd97f0389d5352967b: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:16.451092 20756 tablet_bootstrap.cc:492] T d6253f05a17641889fdd0295a9e1090c P 004e9d427aac45f7a51ded8b76cea407: No bootstrap required, opened a new log
I20260620 02:57:16.451392 20756 ts_tablet_manager.cc:1397] T d6253f05a17641889fdd0295a9e1090c P 004e9d427aac45f7a51ded8b76cea407: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260620 02:57:16.452728 20757 tablet_bootstrap.cc:492] T d6253f05a17641889fdd0295a9e1090c P d1cd48847bdf47bd97f0389d5352967b: No bootstrap required, opened a new log
I20260620 02:57:16.453028 20757 ts_tablet_manager.cc:1397] T d6253f05a17641889fdd0295a9e1090c P d1cd48847bdf47bd97f0389d5352967b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260620 02:57:16.460894 20755 raft_consensus.cc:348] T d6253f05a17641889fdd0295a9e1090c P 0b669241035c42f1b0a1ea7be21bf80d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1cd48847bdf47bd97f0389d5352967b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43923 } } peers { permanent_uuid: "004e9d427aac45f7a51ded8b76cea407" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37575 } } peers { permanent_uuid: "0b669241035c42f1b0a1ea7be21bf80d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46397 } }
I20260620 02:57:16.461400 20755 raft_consensus.cc:374] T d6253f05a17641889fdd0295a9e1090c P 0b669241035c42f1b0a1ea7be21bf80d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:16.461539 20755 raft_consensus.cc:729] T d6253f05a17641889fdd0295a9e1090c P 0b669241035c42f1b0a1ea7be21bf80d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b669241035c42f1b0a1ea7be21bf80d, State: Initialized, Role: FOLLOWER
I20260620 02:57:16.461278 20757 raft_consensus.cc:348] T d6253f05a17641889fdd0295a9e1090c P d1cd48847bdf47bd97f0389d5352967b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1cd48847bdf47bd97f0389d5352967b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43923 } } peers { permanent_uuid: "004e9d427aac45f7a51ded8b76cea407" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37575 } } peers { permanent_uuid: "0b669241035c42f1b0a1ea7be21bf80d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46397 } }
I20260620 02:57:16.461776 20757 raft_consensus.cc:374] T d6253f05a17641889fdd0295a9e1090c P d1cd48847bdf47bd97f0389d5352967b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:16.461956 20757 raft_consensus.cc:729] T d6253f05a17641889fdd0295a9e1090c P d1cd48847bdf47bd97f0389d5352967b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1cd48847bdf47bd97f0389d5352967b, State: Initialized, Role: FOLLOWER
I20260620 02:57:16.461913 20756 raft_consensus.cc:348] T d6253f05a17641889fdd0295a9e1090c P 004e9d427aac45f7a51ded8b76cea407 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1cd48847bdf47bd97f0389d5352967b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43923 } } peers { permanent_uuid: "004e9d427aac45f7a51ded8b76cea407" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37575 } } peers { permanent_uuid: "0b669241035c42f1b0a1ea7be21bf80d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46397 } }
I20260620 02:57:16.462409 20756 raft_consensus.cc:374] T d6253f05a17641889fdd0295a9e1090c P 004e9d427aac45f7a51ded8b76cea407 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:16.462224 20755 consensus_queue.cc:260] T d6253f05a17641889fdd0295a9e1090c P 0b669241035c42f1b0a1ea7be21bf80d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1cd48847bdf47bd97f0389d5352967b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43923 } } peers { permanent_uuid: "004e9d427aac45f7a51ded8b76cea407" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37575 } } peers { permanent_uuid: "0b669241035c42f1b0a1ea7be21bf80d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46397 } }
I20260620 02:57:16.462517 20756 raft_consensus.cc:729] T d6253f05a17641889fdd0295a9e1090c P 004e9d427aac45f7a51ded8b76cea407 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 004e9d427aac45f7a51ded8b76cea407, State: Initialized, Role: FOLLOWER
I20260620 02:57:16.462661 20757 consensus_queue.cc:260] T d6253f05a17641889fdd0295a9e1090c P d1cd48847bdf47bd97f0389d5352967b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1cd48847bdf47bd97f0389d5352967b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43923 } } peers { permanent_uuid: "004e9d427aac45f7a51ded8b76cea407" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37575 } } peers { permanent_uuid: "0b669241035c42f1b0a1ea7be21bf80d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46397 } }
I20260620 02:57:16.463593 20737 heartbeater.cc:503] Master 127.0.97.62:39399 was elected leader, sending a full tablet report...
I20260620 02:57:16.464270 20755 ts_tablet_manager.cc:1428] T d6253f05a17641889fdd0295a9e1090c P 0b669241035c42f1b0a1ea7be21bf80d: Time spent starting tablet: real 0.015s user 0.014s sys 0.002s
I20260620 02:57:16.463328 20756 consensus_queue.cc:260] T d6253f05a17641889fdd0295a9e1090c P 004e9d427aac45f7a51ded8b76cea407 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1cd48847bdf47bd97f0389d5352967b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43923 } } peers { permanent_uuid: "004e9d427aac45f7a51ded8b76cea407" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37575 } } peers { permanent_uuid: "0b669241035c42f1b0a1ea7be21bf80d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46397 } }
I20260620 02:57:16.465914 20757 ts_tablet_manager.cc:1428] T d6253f05a17641889fdd0295a9e1090c P d1cd48847bdf47bd97f0389d5352967b: Time spent starting tablet: real 0.013s user 0.007s sys 0.006s
I20260620 02:57:16.469148 20607 heartbeater.cc:503] Master 127.0.97.62:39399 was elected leader, sending a full tablet report...
I20260620 02:57:16.470731 20756 ts_tablet_manager.cc:1428] T d6253f05a17641889fdd0295a9e1090c P 004e9d427aac45f7a51ded8b76cea407: Time spent starting tablet: real 0.019s user 0.016s sys 0.000s
I20260620 02:57:16.545734 20762 raft_consensus.cc:482] T d6253f05a17641889fdd0295a9e1090c P d1cd48847bdf47bd97f0389d5352967b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 02:57:16.546150 20762 raft_consensus.cc:504] T d6253f05a17641889fdd0295a9e1090c P d1cd48847bdf47bd97f0389d5352967b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1cd48847bdf47bd97f0389d5352967b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43923 } } peers { permanent_uuid: "004e9d427aac45f7a51ded8b76cea407" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37575 } } peers { permanent_uuid: "0b669241035c42f1b0a1ea7be21bf80d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46397 } }
I20260620 02:57:16.548144 20762 leader_election.cc:288] T d6253f05a17641889fdd0295a9e1090c P d1cd48847bdf47bd97f0389d5352967b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 004e9d427aac45f7a51ded8b76cea407 (127.0.97.2:37575), 0b669241035c42f1b0a1ea7be21bf80d (127.0.97.3:46397)
I20260620 02:57:16.555546 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:16.548720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49099 (local address 127.0.97.2:37575)
0620 02:57:16.548960 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:16.548969 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:16.549548 (+ 579us) server_negotiation.cc:407] Connection header received
0620 02:57:16.549755 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:16.549761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:16.549852 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:16.549999 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:57:16.550872 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.551621 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.553724 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.554601 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.554699 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:16.554825 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:16.555141 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 02:57:16.555287 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 02:57:16.555586 20764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:16.548493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37575 (local address 127.0.97.1:49099)
0620 02:57:16.549375 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:16.549401 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:16.549606 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:16.550008 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:16.550012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:16.550030 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:57:16.550671 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:16.550688 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.551764 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.551770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:16.553497 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:16.553513 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.554781 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.554788 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:16.554965 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:16.555006 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 02:57:16.555105 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 02:57:16.555208 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":27,"thread_start_us":436,"threads_started":1}
I20260620 02:57:16.555982 20754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:16.548716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43681 (local address 127.0.97.3:46397)
0620 02:57:16.549053 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:16.549060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:16.549635 (+ 575us) server_negotiation.cc:407] Connection header received
0620 02:57:16.549752 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:16.549757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:16.549841 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:16.549953 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 02:57:16.551017 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.551855 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.553863 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.554840 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.554970 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:16.555062 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:16.555354 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 02:57:16.555485 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260620 02:57:16.556092 20765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:16.548591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46397 (local address 127.0.97.1:43681)
0620 02:57:16.549512 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:16.549528 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:16.549635 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:16.549984 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:16.549991 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:16.550022 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 02:57:16.550792 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:16.550811 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.552089 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.552097 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:16.553687 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:16.553704 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.555020 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.555025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:16.555187 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:16.555222 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 02:57:16.555314 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 02:57:16.555452 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"mutex_wait_us":76,"thread_start_us":572,"threads_started":1}
I20260620 02:57:16.556772 20562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6253f05a17641889fdd0295a9e1090c" candidate_uuid: "d1cd48847bdf47bd97f0389d5352967b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "004e9d427aac45f7a51ded8b76cea407" is_pre_election: true
I20260620 02:57:16.557186 20692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6253f05a17641889fdd0295a9e1090c" candidate_uuid: "d1cd48847bdf47bd97f0389d5352967b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b669241035c42f1b0a1ea7be21bf80d" is_pre_election: true
I20260620 02:57:16.557322 20562 raft_consensus.cc:2454] T d6253f05a17641889fdd0295a9e1090c P 004e9d427aac45f7a51ded8b76cea407 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1cd48847bdf47bd97f0389d5352967b in term 0.
I20260620 02:57:16.557705 20692 raft_consensus.cc:2454] T d6253f05a17641889fdd0295a9e1090c P 0b669241035c42f1b0a1ea7be21bf80d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1cd48847bdf47bd97f0389d5352967b in term 0.
I20260620 02:57:16.558034 20367 leader_election.cc:302] T d6253f05a17641889fdd0295a9e1090c P d1cd48847bdf47bd97f0389d5352967b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 004e9d427aac45f7a51ded8b76cea407, d1cd48847bdf47bd97f0389d5352967b; no voters:
I20260620 02:57:16.558480 20762 raft_consensus.cc:2781] T d6253f05a17641889fdd0295a9e1090c P d1cd48847bdf47bd97f0389d5352967b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 02:57:16.558661 20762 raft_consensus.cc:482] T d6253f05a17641889fdd0295a9e1090c P d1cd48847bdf47bd97f0389d5352967b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 02:57:16.558779 20762 raft_consensus.cc:3037] T d6253f05a17641889fdd0295a9e1090c P d1cd48847bdf47bd97f0389d5352967b [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:16.560940 20762 raft_consensus.cc:504] T d6253f05a17641889fdd0295a9e1090c P d1cd48847bdf47bd97f0389d5352967b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1cd48847bdf47bd97f0389d5352967b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43923 } } peers { permanent_uuid: "004e9d427aac45f7a51ded8b76cea407" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37575 } } peers { permanent_uuid: "0b669241035c42f1b0a1ea7be21bf80d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46397 } }
I20260620 02:57:16.561583 20762 leader_election.cc:288] T d6253f05a17641889fdd0295a9e1090c P d1cd48847bdf47bd97f0389d5352967b [CANDIDATE]: Term 1 election: Requested vote from peers 004e9d427aac45f7a51ded8b76cea407 (127.0.97.2:37575), 0b669241035c42f1b0a1ea7be21bf80d (127.0.97.3:46397)
I20260620 02:57:16.562201 20562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6253f05a17641889fdd0295a9e1090c" candidate_uuid: "d1cd48847bdf47bd97f0389d5352967b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "004e9d427aac45f7a51ded8b76cea407"
I20260620 02:57:16.562201 20692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6253f05a17641889fdd0295a9e1090c" candidate_uuid: "d1cd48847bdf47bd97f0389d5352967b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b669241035c42f1b0a1ea7be21bf80d"
I20260620 02:57:16.562446 20562 raft_consensus.cc:3037] T d6253f05a17641889fdd0295a9e1090c P 004e9d427aac45f7a51ded8b76cea407 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:16.562480 20692 raft_consensus.cc:3037] T d6253f05a17641889fdd0295a9e1090c P 0b669241035c42f1b0a1ea7be21bf80d [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:16.565775 20692 raft_consensus.cc:2454] T d6253f05a17641889fdd0295a9e1090c P 0b669241035c42f1b0a1ea7be21bf80d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1cd48847bdf47bd97f0389d5352967b in term 1.
I20260620 02:57:16.565778 20562 raft_consensus.cc:2454] T d6253f05a17641889fdd0295a9e1090c P 004e9d427aac45f7a51ded8b76cea407 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1cd48847bdf47bd97f0389d5352967b in term 1.
I20260620 02:57:16.566442 20367 leader_election.cc:302] T d6253f05a17641889fdd0295a9e1090c P d1cd48847bdf47bd97f0389d5352967b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 004e9d427aac45f7a51ded8b76cea407, 0b669241035c42f1b0a1ea7be21bf80d, d1cd48847bdf47bd97f0389d5352967b; no voters:
I20260620 02:57:16.566798 20762 raft_consensus.cc:2781] T d6253f05a17641889fdd0295a9e1090c P d1cd48847bdf47bd97f0389d5352967b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:57:16.567271 20762 raft_consensus.cc:686] T d6253f05a17641889fdd0295a9e1090c P d1cd48847bdf47bd97f0389d5352967b [term 1 LEADER]: Becoming Leader. State: Replica: d1cd48847bdf47bd97f0389d5352967b, State: Running, Role: LEADER
I20260620 02:57:16.567744 20762 consensus_queue.cc:237] T d6253f05a17641889fdd0295a9e1090c P d1cd48847bdf47bd97f0389d5352967b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1cd48847bdf47bd97f0389d5352967b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43923 } } peers { permanent_uuid: "004e9d427aac45f7a51ded8b76cea407" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37575 } } peers { permanent_uuid: "0b669241035c42f1b0a1ea7be21bf80d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46397 } }
I20260620 02:57:16.572548 20288 catalog_manager.cc:5495] T d6253f05a17641889fdd0295a9e1090c P d1cd48847bdf47bd97f0389d5352967b reported cstate change: term changed from 0 to 1, leader changed from <none> to d1cd48847bdf47bd97f0389d5352967b (127.0.97.1). New cstate: current_term: 1 leader_uuid: "d1cd48847bdf47bd97f0389d5352967b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1cd48847bdf47bd97f0389d5352967b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "004e9d427aac45f7a51ded8b76cea407" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b669241035c42f1b0a1ea7be21bf80d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46397 } health_report { overall_health: UNKNOWN } } }
W20260620 02:57:16.580237 20738 tablet.cc:2369] T d6253f05a17641889fdd0295a9e1090c P 0b669241035c42f1b0a1ea7be21bf80d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 02:57:16.596622 20608 tablet.cc:2369] T d6253f05a17641889fdd0295a9e1090c P 004e9d427aac45f7a51ded8b76cea407: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:57:16.619400 20750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:16.613691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47290 (local address 127.0.97.1:43923)
0620 02:57:16.613880 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:16.613884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:16.613927 (+ 43us) server_negotiation.cc:407] Connection header received
0620 02:57:16.614178 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:16.614182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:16.614280 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:16.614515 (+ 235us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:57:16.614864 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.615679 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.617177 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.617601 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.617706 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:16.618221 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:57:16.618834 (+ 613us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:57:16.618969 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:16.619132 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 02:57:16.619204 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 02:57:16.628139 20692 raft_consensus.cc:1261] T d6253f05a17641889fdd0295a9e1090c P 0b669241035c42f1b0a1ea7be21bf80d [term 1 FOLLOWER]: Refusing update from remote peer d1cd48847bdf47bd97f0389d5352967b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:57:16.628139 20562 raft_consensus.cc:1261] T d6253f05a17641889fdd0295a9e1090c P 004e9d427aac45f7a51ded8b76cea407 [term 1 FOLLOWER]: Refusing update from remote peer d1cd48847bdf47bd97f0389d5352967b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:57:16.629266 20766 consensus_queue.cc:1035] T d6253f05a17641889fdd0295a9e1090c P d1cd48847bdf47bd97f0389d5352967b [LEADER]: Connected to new peer: Peer: permanent_uuid: "004e9d427aac45f7a51ded8b76cea407" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:57:16.629675 20762 consensus_queue.cc:1035] T d6253f05a17641889fdd0295a9e1090c P d1cd48847bdf47bd97f0389d5352967b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b669241035c42f1b0a1ea7be21bf80d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 02:57:16.630889 20477 tablet.cc:2369] T d6253f05a17641889fdd0295a9e1090c P d1cd48847bdf47bd97f0389d5352967b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:57:16.642477 20769 mvcc.cc:205] Tried to move back new op lower bound from 7298761673217544192 to 7298761672986378240. Current Snapshot: MvccSnapshot[applied={T|T < 7298761673217544192}]
I20260620 02:57:16.645094 20770 mvcc.cc:205] Tried to move back new op lower bound from 7298761673217544192 to 7298761672986378240. Current Snapshot: MvccSnapshot[applied={T|T < 7298761673217544192}]
I20260620 02:57:16.659566 20289 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:57:16.659910 20289 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 02:57:16.663990 20289 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4fbf36b122643b6b60a0ee476dbc278: Sending DeleteTablet for 3 replicas of tablet d6253f05a17641889fdd0295a9e1090c
I20260620 02:57:16.665307 20542 tablet_service.cc:1514] Processing DeleteTablet for tablet d6253f05a17641889fdd0295a9e1090c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:57:16 UTC) from {username='slave'} at 127.0.0.1:58796
I20260620 02:57:16.665310 20411 tablet_service.cc:1514] Processing DeleteTablet for tablet d6253f05a17641889fdd0295a9e1090c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:57:16 UTC) from {username='slave'} at 127.0.0.1:47274
I20260620 02:57:16.665334 20672 tablet_service.cc:1514] Processing DeleteTablet for tablet d6253f05a17641889fdd0295a9e1090c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 02:57:16 UTC) from {username='slave'} at 127.0.0.1:55086
I20260620 02:57:16.667797 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 20349
I20260620 02:57:16.677771 20780 tablet_replica.cc:330] T d6253f05a17641889fdd0295a9e1090c P d1cd48847bdf47bd97f0389d5352967b: stopping tablet replica
I20260620 02:57:16.678999 20781 tablet_replica.cc:330] T d6253f05a17641889fdd0295a9e1090c P 0b669241035c42f1b0a1ea7be21bf80d: stopping tablet replica
I20260620 02:57:16.679564 20781 raft_consensus.cc:2229] T d6253f05a17641889fdd0295a9e1090c P 0b669241035c42f1b0a1ea7be21bf80d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:57:16.680241 20781 raft_consensus.cc:2258] T d6253f05a17641889fdd0295a9e1090c P 0b669241035c42f1b0a1ea7be21bf80d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:57:16.680698 20780 raft_consensus.cc:2229] T d6253f05a17641889fdd0295a9e1090c P d1cd48847bdf47bd97f0389d5352967b [term 1 LEADER]: Raft consensus shutting down.
I20260620 02:57:16.681675 20780 raft_consensus.cc:2258] T d6253f05a17641889fdd0295a9e1090c P d1cd48847bdf47bd97f0389d5352967b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:57:16.682570 20782 tablet_replica.cc:330] T d6253f05a17641889fdd0295a9e1090c P 004e9d427aac45f7a51ded8b76cea407: stopping tablet replica
I20260620 02:57:16.683133 20782 raft_consensus.cc:2229] T d6253f05a17641889fdd0295a9e1090c P 004e9d427aac45f7a51ded8b76cea407 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 02:57:16.683703 20782 raft_consensus.cc:2258] T d6253f05a17641889fdd0295a9e1090c P 004e9d427aac45f7a51ded8b76cea407 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 02:57:16.685400 20781 ts_tablet_manager.cc:1905] T d6253f05a17641889fdd0295a9e1090c P 0b669241035c42f1b0a1ea7be21bf80d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:57:16.686036 20782 ts_tablet_manager.cc:1905] T d6253f05a17641889fdd0295a9e1090c P 004e9d427aac45f7a51ded8b76cea407: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:57:16.692065 20750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:16.669505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47298 (local address 127.0.97.1:43923)
0620 02:57:16.669737 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:16.669746 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:16.683412 (+ 13666us) server_negotiation.cc:407] Connection header received
0620 02:57:16.683901 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:16.683906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:16.684011 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:16.684109 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:16.684798 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.685999 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.687918 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.688584 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.688677 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:16.690212 (+ 1535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:16.690224 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:16.690241 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:16.691034 (+ 793us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:16.691102 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:16.691126 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:16.691227 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:16.691481 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 02:57:16.691699 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 02:57:16.692708 20781 ts_tablet_manager.cc:1918] T d6253f05a17641889fdd0295a9e1090c P 0b669241035c42f1b0a1ea7be21bf80d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:57:16.692896 20781 log.cc:1196] T d6253f05a17641889fdd0295a9e1090c P 0b669241035c42f1b0a1ea7be21bf80d: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-2/wal/wals/d6253f05a17641889fdd0295a9e1090c
I20260620 02:57:16.693367 20471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:57:16.693835 20782 ts_tablet_manager.cc:1918] T d6253f05a17641889fdd0295a9e1090c P 004e9d427aac45f7a51ded8b76cea407: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 02:57:16.694177 20782 log.cc:1196] T d6253f05a17641889fdd0295a9e1090c P 004e9d427aac45f7a51ded8b76cea407: Deleting WAL directory at /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781923914418765-388-0/minicluster-data/ts-1/wal/wals/d6253f05a17641889fdd0295a9e1090c
I20260620 02:57:16.695098 20782 ts_tablet_manager.cc:1939] T d6253f05a17641889fdd0295a9e1090c P 004e9d427aac45f7a51ded8b76cea407: Deleting consensus metadata
I20260620 02:57:16.695333 20781 ts_tablet_manager.cc:1939] T d6253f05a17641889fdd0295a9e1090c P 0b669241035c42f1b0a1ea7be21bf80d: Deleting consensus metadata
I20260620 02:57:16.697280 20277 catalog_manager.cc:4841] TS 004e9d427aac45f7a51ded8b76cea407 (127.0.97.2:37575): tablet d6253f05a17641889fdd0295a9e1090c (table test-table [id=d6ac77be8306411e924839496ab626b6]) successfully deleted
I20260620 02:57:16.698278 20274 catalog_manager.cc:4841] TS 0b669241035c42f1b0a1ea7be21bf80d (127.0.97.3:46397): tablet d6253f05a17641889fdd0295a9e1090c (table test-table [id=d6ac77be8306411e924839496ab626b6]) successfully deleted
I20260620 02:57:16.694258 20780 ts_tablet_manager.cc:1905] T d6253f05a17641889fdd0295a9e1090c P d1cd48847bdf47bd97f0389d5352967b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 02:57:16.797091 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 20349
I20260620 02:57:16.816756 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 20480
W20260620 02:57:16.816790 20275 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:43923 (error 108)
I20260620 02:57:16.818702 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:16.818256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43923 (local address 127.0.0.1:47310)
0620 02:57:16.818471 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:16.818567 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 02:57:16.819000 20275 catalog_manager.cc:4584] TS d1cd48847bdf47bd97f0389d5352967b (127.0.97.1:43923): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d6253f05a17641889fdd0295a9e1090c: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43923: connect: Connection refused (error 111)
I20260620 02:57:16.824322 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:16.817617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58798 (local address 127.0.97.2:37575)
0620 02:57:16.817803 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:16.817809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:16.817933 (+ 124us) server_negotiation.cc:407] Connection header received
0620 02:57:16.818180 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:16.818186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:16.818275 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:16.818400 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:16.818803 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.819603 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.820863 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.821260 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.821384 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:16.822502 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:16.822541 (+ 39us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:16.822554 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:16.823371 (+ 817us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:16.823433 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:16.823458 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:16.823578 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:16.823863 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 02:57:16.823976 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 02:57:16.825094 20602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:57:16.872661 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:16.872195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43923 (local address 127.0.0.1:47318)
0620 02:57:16.872406 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:16.872500 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 02:57:16.873720 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:16.873383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43923 (local address 127.0.0.1:47330)
0620 02:57:16.873522 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:16.873594 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 02:57:16.922950 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 20480
I20260620 02:57:16.934521 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:16.933866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43923 (local address 127.0.0.1:47336)
0620 02:57:16.934140 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:16.934265 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 02:57:16.935667 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:16.935232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43923 (local address 127.0.0.1:47344)
0620 02:57:16.935410 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:16.935508 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 02:57:16.940233 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 20610
I20260620 02:57:16.946949 20754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:16.940955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55092 (local address 127.0.97.3:46397)
0620 02:57:16.941156 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:16.941161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:16.941179 (+ 18us) server_negotiation.cc:407] Connection header received
0620 02:57:16.941276 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:16.941280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:16.941362 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:16.941471 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:16.941855 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.942692 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.944040 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:16.944409 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:16.944495 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:16.945602 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:16.945610 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:16.945625 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:16.946211 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:16.946258 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:16.946277 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:16.946508 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:16.946614 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 02:57:16.946716 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 02:57:16.947605 20732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:57:17.016566 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:17.016032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43923 (local address 127.0.0.1:47348)
0620 02:57:17.016292 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:17.016400 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260620 02:57:17.017653 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:17.017272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43923 (local address 127.0.0.1:47356)
0620 02:57:17.017466 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:17.017534 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 02:57:17.037621 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 20610
I20260620 02:57:17.056085 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 20259
I20260620 02:57:17.057142 20320 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:57:17.169178 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 20259
2026-06-20T02:57:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2859 ms)
[----------] 6 tests from EncryptionPolicyTest (22465 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:57:17 dist-test-slave-8p8c krb5kdc[20791](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:57:17 dist-test-slave-8p8c krb5kdc[20791](info): set up 2 sockets
Jun 20 02:57:17 dist-test-slave-8p8c krb5kdc[20791](info): commencing operation
krb5kdc: starting...
W20260620 02:57:18.314824 388 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:57:18 dist-test-slave-8p8c krb5kdc[20791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:57:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:57:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:57:18.570832 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35947
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:57:18.905678 20804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:18.905977 20804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:18.906136 20804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:18.913503 20804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:57:18.913630 20804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:18.913730 20804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:18.913825 20804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:57:18.913919 20804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:57:18.924280 20804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35947
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:18.925647 20804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:18.927074 20804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:18.939179 20810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:18.939245 20812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:18.939220 20809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:18.940205 20804 server_base.cc:1034] running on GCE node
I20260620 02:57:18.940932 20804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:18.942278 20804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:18.943540 20804 hybrid_clock.cc:648] HybridClock initialized: now 1781924238943487 us; error 45 us; skew 500 ppm
I20260620 02:57:18.943939 20804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:57:18 dist-test-slave-8p8c krb5kdc[20791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:57:18.952689 20804 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:57:18.954478 20804 webserver.cc:460] Webserver started at http://127.0.0.1:35159/ using document root <none> and password file <none>
I20260620 02:57:18.955221 20804 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:18.955361 20804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:18.955685 20804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:18.958841 20804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "100fbf778a5645dda58fafdc3fe3ca38"
format_stamp: "Formatted at 2026-06-20 02:57:18 on dist-test-slave-8p8c"
server_key: "dfa8954c4e8b56ca4b97122745c752d0"
server_key_iv: "e5d3b66a026bc6b1a503ee95addcf9f6"
server_key_version: "encryptionkey@0"
I20260620 02:57:18.959646 20804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "100fbf778a5645dda58fafdc3fe3ca38"
format_stamp: "Formatted at 2026-06-20 02:57:18 on dist-test-slave-8p8c"
server_key: "dfa8954c4e8b56ca4b97122745c752d0"
server_key_iv: "e5d3b66a026bc6b1a503ee95addcf9f6"
server_key_version: "encryptionkey@0"
I20260620 02:57:18.965040 20804 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:57:18.968880 20819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:18.970338 20804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:57:18.970602 20804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "100fbf778a5645dda58fafdc3fe3ca38"
format_stamp: "Formatted at 2026-06-20 02:57:18 on dist-test-slave-8p8c"
server_key: "dfa8954c4e8b56ca4b97122745c752d0"
server_key_iv: "e5d3b66a026bc6b1a503ee95addcf9f6"
server_key_version: "encryptionkey@0"
I20260620 02:57:18.970811 20804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:18.988952 20804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:18.993798 20804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:18.994114 20804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:19.014477 20804 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35947
I20260620 02:57:19.014514 20870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35947 every 8 connection(s)
I20260620 02:57:19.015883 20804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:57:19.019637 20871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:19.026113 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 20804
I20260620 02:57:19.026386 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:57:19.027249 388 external_mini_cluster.cc:1428] Setting key f582bf6664a17ce061bd380d6fed78fa
I20260620 02:57:19.030988 20871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 100fbf778a5645dda58fafdc3fe3ca38: Bootstrap starting.
I20260620 02:57:19.035286 20871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 100fbf778a5645dda58fafdc3fe3ca38: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:19.036535 20871 log.cc:824] T 00000000000000000000000000000000 P 100fbf778a5645dda58fafdc3fe3ca38: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:19.039649 20871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 100fbf778a5645dda58fafdc3fe3ca38: No bootstrap required, opened a new log
Jun 20 02:57:19 dist-test-slave-8p8c krb5kdc[20791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 02:57:19.049815 20871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 100fbf778a5645dda58fafdc3fe3ca38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100fbf778a5645dda58fafdc3fe3ca38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35947 } }
I20260620 02:57:19.050413 20871 raft_consensus.cc:374] T 00000000000000000000000000000000 P 100fbf778a5645dda58fafdc3fe3ca38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:19.050544 20871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 100fbf778a5645dda58fafdc3fe3ca38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100fbf778a5645dda58fafdc3fe3ca38, State: Initialized, Role: FOLLOWER
I20260620 02:57:19.051405 20871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 100fbf778a5645dda58fafdc3fe3ca38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100fbf778a5645dda58fafdc3fe3ca38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35947 } }
I20260620 02:57:19.051870 20871 raft_consensus.cc:388] T 00000000000000000000000000000000 P 100fbf778a5645dda58fafdc3fe3ca38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:57:19.052047 20871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 100fbf778a5645dda58fafdc3fe3ca38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:57:19.052239 20871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 100fbf778a5645dda58fafdc3fe3ca38 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:19.055992 20871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 100fbf778a5645dda58fafdc3fe3ca38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100fbf778a5645dda58fafdc3fe3ca38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35947 } }
I20260620 02:57:19.056681 20871 leader_election.cc:302] T 00000000000000000000000000000000 P 100fbf778a5645dda58fafdc3fe3ca38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 100fbf778a5645dda58fafdc3fe3ca38; no voters:
I20260620 02:57:19.057271 20874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:19.029646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58277 (local address 127.0.0.1:35947)
0620 02:57:19.030696 (+ 1050us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:19.030713 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:19.031101 (+ 388us) server_negotiation.cc:407] Connection header received
0620 02:57:19.033778 (+ 2677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:19.033807 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:19.034345 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:19.034849 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:19.036307 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:19.038104 (+ 1797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:19.039497 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:19.040606 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:19.040831 (+ 225us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:19.045960 (+ 5129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:19.045977 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:19.046004 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:19.046063 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:19.051170 (+ 5107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:19.052607 (+ 1437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:19.052620 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:19.052662 (+ 42us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:19.052896 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:19.053451 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:19.053457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:19.053461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:19.054257 (+ 796us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:19.054745 (+ 488us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:19.055117 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 02:57:19.055561 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":451,"threads_started":1}
I20260620 02:57:19.057466 20871 leader_election.cc:288] T 00000000000000000000000000000000 P 100fbf778a5645dda58fafdc3fe3ca38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:57:19.057619 20876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 100fbf778a5645dda58fafdc3fe3ca38 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:57:19.059752 20876 raft_consensus.cc:686] T 00000000000000000000000000000000 P 100fbf778a5645dda58fafdc3fe3ca38 [term 1 LEADER]: Becoming Leader. State: Replica: 100fbf778a5645dda58fafdc3fe3ca38, State: Running, Role: LEADER
I20260620 02:57:19.060016 20871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 100fbf778a5645dda58fafdc3fe3ca38 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:57:19.060343 20876 consensus_queue.cc:237] T 00000000000000000000000000000000 P 100fbf778a5645dda58fafdc3fe3ca38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100fbf778a5645dda58fafdc3fe3ca38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35947 } }
I20260620 02:57:19.062870 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:57:19.063966 20878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 100fbf778a5645dda58fafdc3fe3ca38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 100fbf778a5645dda58fafdc3fe3ca38. Latest consensus state: current_term: 1 leader_uuid: "100fbf778a5645dda58fafdc3fe3ca38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100fbf778a5645dda58fafdc3fe3ca38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35947 } } }
I20260620 02:57:19.064402 20878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 100fbf778a5645dda58fafdc3fe3ca38 [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:19.063997 20877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 100fbf778a5645dda58fafdc3fe3ca38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "100fbf778a5645dda58fafdc3fe3ca38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100fbf778a5645dda58fafdc3fe3ca38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35947 } } }
I20260620 02:57:19.065069 20877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 100fbf778a5645dda58fafdc3fe3ca38 [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:19.068733 20884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:57:19.075011 20884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:57:19.089432 20884 catalog_manager.cc:1347] Generated new cluster ID: 0fe65074f9da44cbbc77a3ef79ce74dc
I20260620 02:57:19.089546 20884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:57:19.089985 20874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:19.071284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37887 (local address 127.0.0.1:35947)
0620 02:57:19.071484 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:19.071490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:19.072033 (+ 543us) server_negotiation.cc:407] Connection header received
0620 02:57:19.072321 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:19.072329 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:19.072410 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:19.072495 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:19.074665 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:19.075890 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:19.077839 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:19.078578 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:19.078673 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:19.082128 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:19.082136 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:19.082141 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:19.082184 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:19.086788 (+ 4604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:19.087579 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:19.087584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:19.087591 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:19.087728 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:19.088277 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:19.088282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:19.088285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:19.088692 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:19.088965 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:19.089393 (+ 428us) server_negotiation.cc:299] Negotiation successful
0620 02:57:19.089598 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 02:57:19.147723 20884 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:57:19.149056 20884 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:57:19.155725 20884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 100fbf778a5645dda58fafdc3fe3ca38: Generated new TSK 0
I20260620 02:57:19.156832 20884 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:57:19.162545 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 20804
I20260620 02:57:19.164350 20866 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:57:19.259328 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 20804
W20260620 02:57:19.276584 20789 connection.cc:535] client connection to 127.0.0.1:35947 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35947): Connection reset by peer (error 104)
2026-06-20T02:57:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2079 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:57:19 dist-test-slave-8p8c krb5kdc[20903](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:57:19 dist-test-slave-8p8c krb5kdc[20903](info): set up 2 sockets
Jun 20 02:57:19 dist-test-slave-8p8c krb5kdc[20903](info): commencing operation
krb5kdc: starting...
W20260620 02:57:20.358892 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:57:20 dist-test-slave-8p8c krb5kdc[20903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:57:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:57:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:57:20.614019 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33269
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:57:20.953624 20916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:20.953953 20916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:20.954110 20916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:20.961992 20916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:57:20.962134 20916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:20.962222 20916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:20.962328 20916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:57:20.962420 20916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:57:20.972695 20916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33269
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:20.974097 20916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:20.975492 20916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:20.988205 20924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:20.988238 20921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:20.988286 20922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:20.989142 20916 server_base.cc:1034] running on GCE node
I20260620 02:57:20.990195 20916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:20.991824 20916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:20.993037 20916 hybrid_clock.cc:648] HybridClock initialized: now 1781924240992951 us; error 63 us; skew 500 ppm
I20260620 02:57:20.993538 20916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:57:20 dist-test-slave-8p8c krb5kdc[20903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:57:20.997109 20916 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:57:20.999171 20916 webserver.cc:460] Webserver started at http://127.0.0.1:46145/ using document root <none> and password file <none>
I20260620 02:57:20.999934 20916 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:21.000116 20916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:21.000516 20916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:21.003912 20916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "5f7fe3cec8a14cf8ba30e25e132ad284"
format_stamp: "Formatted at 2026-06-20 02:57:20 on dist-test-slave-8p8c"
server_key: "dbaf0fd1e147b334539ed3f8d4c3d985"
server_key_iv: "74ac6c2ef0e5fbb8e9be97839818c433"
server_key_version: "encryptionkey@0"
I20260620 02:57:21.004720 20916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "5f7fe3cec8a14cf8ba30e25e132ad284"
format_stamp: "Formatted at 2026-06-20 02:57:20 on dist-test-slave-8p8c"
server_key: "dbaf0fd1e147b334539ed3f8d4c3d985"
server_key_iv: "74ac6c2ef0e5fbb8e9be97839818c433"
server_key_version: "encryptionkey@0"
I20260620 02:57:21.010054 20916 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:57:21.013692 20931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:21.015430 20916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:57:21.015683 20916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "5f7fe3cec8a14cf8ba30e25e132ad284"
format_stamp: "Formatted at 2026-06-20 02:57:20 on dist-test-slave-8p8c"
server_key: "dbaf0fd1e147b334539ed3f8d4c3d985"
server_key_iv: "74ac6c2ef0e5fbb8e9be97839818c433"
server_key_version: "encryptionkey@0"
I20260620 02:57:21.015908 20916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:21.029111 20916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:21.033385 20916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:21.033685 20916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:21.054212 20916 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33269
I20260620 02:57:21.054258 20982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33269 every 8 connection(s)
I20260620 02:57:21.055764 20916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:57:21.059020 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 20916
I20260620 02:57:21.059231 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:57:21.059791 388 external_mini_cluster.cc:1428] Setting key f18525fbcb6d991e79b4f9d2fee9f3af
I20260620 02:57:21.060312 20983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:21.074888 20986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:21.061679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42147 (local address 127.0.0.1:33269)
0620 02:57:21.062565 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:21.062577 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:21.062618 (+ 41us) server_negotiation.cc:407] Connection header received
0620 02:57:21.065028 (+ 2410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:21.065052 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:21.065518 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:21.065867 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:21.066497 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:21.067941 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:21.069851 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:21.070700 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:21.070839 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:21.072233 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:21.072242 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:21.072257 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:21.072867 (+ 610us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:21.072948 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:21.072973 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:21.073098 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:21.073505 (+ 407us) server_negotiation.cc:299] Negotiation successful
0620 02:57:21.073711 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":650,"thread_start_us":436,"threads_started":1}
I20260620 02:57:21.074992 20983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f7fe3cec8a14cf8ba30e25e132ad284: Bootstrap starting.
I20260620 02:57:21.078297 20983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f7fe3cec8a14cf8ba30e25e132ad284: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:21.079100 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:57:21.079612 20983 log.cc:824] T 00000000000000000000000000000000 P 5f7fe3cec8a14cf8ba30e25e132ad284: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:21.083304 20983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f7fe3cec8a14cf8ba30e25e132ad284: No bootstrap required, opened a new log
I20260620 02:57:21.091588 20986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:21.084212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60777 (local address 127.0.0.1:33269)
0620 02:57:21.084424 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:21.084431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:21.084944 (+ 513us) server_negotiation.cc:407] Connection header received
0620 02:57:21.085034 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:21.085038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:21.085144 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:21.085235 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:21.085840 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:21.086760 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:21.088742 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:21.089293 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:21.089406 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:21.090761 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:21.090766 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:21.090770 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:21.090789 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:21.090835 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:21.090853 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:21.090949 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:21.091175 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 02:57:21.091250 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 02:57:21.093477 20983 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f7fe3cec8a14cf8ba30e25e132ad284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7fe3cec8a14cf8ba30e25e132ad284" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33269 } }
I20260620 02:57:21.093884 20983 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f7fe3cec8a14cf8ba30e25e132ad284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:21.094000 20983 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f7fe3cec8a14cf8ba30e25e132ad284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f7fe3cec8a14cf8ba30e25e132ad284, State: Initialized, Role: FOLLOWER
I20260620 02:57:21.094658 20983 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f7fe3cec8a14cf8ba30e25e132ad284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7fe3cec8a14cf8ba30e25e132ad284" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33269 } }
I20260620 02:57:21.094900 20983 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f7fe3cec8a14cf8ba30e25e132ad284 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:57:21.095067 20983 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f7fe3cec8a14cf8ba30e25e132ad284 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:57:21.095232 20983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f7fe3cec8a14cf8ba30e25e132ad284 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:21.097486 20983 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f7fe3cec8a14cf8ba30e25e132ad284 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7fe3cec8a14cf8ba30e25e132ad284" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33269 } }
I20260620 02:57:21.098021 20983 leader_election.cc:302] T 00000000000000000000000000000000 P 5f7fe3cec8a14cf8ba30e25e132ad284 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f7fe3cec8a14cf8ba30e25e132ad284; no voters:
I20260620 02:57:21.098629 20983 leader_election.cc:288] T 00000000000000000000000000000000 P 5f7fe3cec8a14cf8ba30e25e132ad284 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:57:21.098773 20993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f7fe3cec8a14cf8ba30e25e132ad284 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:57:21.099462 20993 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f7fe3cec8a14cf8ba30e25e132ad284 [term 1 LEADER]: Becoming Leader. State: Replica: 5f7fe3cec8a14cf8ba30e25e132ad284, State: Running, Role: LEADER
I20260620 02:57:21.100035 20993 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f7fe3cec8a14cf8ba30e25e132ad284 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7fe3cec8a14cf8ba30e25e132ad284" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33269 } }
I20260620 02:57:21.100724 20983 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f7fe3cec8a14cf8ba30e25e132ad284 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:57:21.103188 20994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f7fe3cec8a14cf8ba30e25e132ad284 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f7fe3cec8a14cf8ba30e25e132ad284" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7fe3cec8a14cf8ba30e25e132ad284" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33269 } } }
I20260620 02:57:21.103667 20994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f7fe3cec8a14cf8ba30e25e132ad284 [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:21.103305 20995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f7fe3cec8a14cf8ba30e25e132ad284 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f7fe3cec8a14cf8ba30e25e132ad284. Latest consensus state: current_term: 1 leader_uuid: "5f7fe3cec8a14cf8ba30e25e132ad284" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7fe3cec8a14cf8ba30e25e132ad284" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33269 } } }
I20260620 02:57:21.105350 20995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f7fe3cec8a14cf8ba30e25e132ad284 [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:21.107156 21001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:57:21.112095 21001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:57:21.120501 21001 catalog_manager.cc:1347] Generated new cluster ID: 7f18374f22a34eceafe278a51161caa2
I20260620 02:57:21.120651 21001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:57:21.132910 21001 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:57:21.134299 21001 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:57:21.141299 21001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f7fe3cec8a14cf8ba30e25e132ad284: Generated new TSK 0
I20260620 02:57:21.142205 21001 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:57:21.156870 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 20916
W20260620 02:57:21.158107 20978 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42147
W20260620 02:57:21.158939 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 20916: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:57:21.159147 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 20916
W20260620 02:57:21.177661 20901 connection.cc:535] client connection to 127.0.0.1:33269 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33269): Connection reset by peer (error 104)
2026-06-20T02:57:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1902 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:57:21 dist-test-slave-8p8c krb5kdc[21014](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:57:21 dist-test-slave-8p8c krb5kdc[21014](info): set up 2 sockets
Jun 20 02:57:21 dist-test-slave-8p8c krb5kdc[21014](info): commencing operation
krb5kdc: starting...
W20260620 02:57:22.282789 388 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:57:22 dist-test-slave-8p8c krb5kdc[21014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:57:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:57:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:57:22.537891 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35755
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:57:22.880811 21027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:22.881191 21027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:22.881356 21027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:22.888753 21027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:57:22.889029 21027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:22.889154 21027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:22.889276 21027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:57:22.889365 21027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:57:22.899770 21027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35755
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:22.901178 21027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:22.902570 21027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:22.915007 21032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:22.915174 21035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:22.915058 21033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:22.915953 21027 server_base.cc:1034] running on GCE node
I20260620 02:57:22.916735 21027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:22.918215 21027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:22.919476 21027 hybrid_clock.cc:648] HybridClock initialized: now 1781924242919406 us; error 60 us; skew 500 ppm
I20260620 02:57:22.919920 21027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:57:22 dist-test-slave-8p8c krb5kdc[21014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:57:22.923730 21027 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:57:22.925356 21027 webserver.cc:460] Webserver started at http://127.0.0.1:34567/ using document root <none> and password file <none>
I20260620 02:57:22.926082 21027 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:22.926196 21027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:22.926569 21027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:22.929893 21027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "dad06b2526274022acaa8af702c05b54"
format_stamp: "Formatted at 2026-06-20 02:57:22 on dist-test-slave-8p8c"
server_key: "4eafee0f2a7af2ba501bd0c8f04195fd"
server_key_iv: "3052493a9037c719d57ae4b12c8d3df7"
server_key_version: "encryptionkey@0"
I20260620 02:57:22.930760 21027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "dad06b2526274022acaa8af702c05b54"
format_stamp: "Formatted at 2026-06-20 02:57:22 on dist-test-slave-8p8c"
server_key: "4eafee0f2a7af2ba501bd0c8f04195fd"
server_key_iv: "3052493a9037c719d57ae4b12c8d3df7"
server_key_version: "encryptionkey@0"
I20260620 02:57:22.936522 21027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:57:22.940486 21042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:22.942070 21027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:57:22.942298 21027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "dad06b2526274022acaa8af702c05b54"
format_stamp: "Formatted at 2026-06-20 02:57:22 on dist-test-slave-8p8c"
server_key: "4eafee0f2a7af2ba501bd0c8f04195fd"
server_key_iv: "3052493a9037c719d57ae4b12c8d3df7"
server_key_version: "encryptionkey@0"
I20260620 02:57:22.942544 21027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:22.948843 21027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:22.953483 21027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:22.953805 21027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:22.973593 21027 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35755
I20260620 02:57:22.973652 21093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35755 every 8 connection(s)
I20260620 02:57:22.975203 21027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:57:22.979280 21094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:22.982339 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 21027
I20260620 02:57:22.982585 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:57:22.983027 388 external_mini_cluster.cc:1428] Setting key 6485c4250050d8907a31fae2da6bbfd7
I20260620 02:57:22.990724 21097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:22.984593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49179 (local address 127.0.0.1:35755)
0620 02:57:22.985418 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:22.985429 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:22.985468 (+ 39us) server_negotiation.cc:407] Connection header received
0620 02:57:22.987231 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:22.987256 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:22.987597 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:22.987935 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:22.989026 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:22.989032 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:22.989048 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:22.989062 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:22.989123 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:22.989194 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:22.989369 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:22.989478 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 02:57:22.989683 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":386,"threads_started":1}
I20260620 02:57:22.992202 21094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dad06b2526274022acaa8af702c05b54: Bootstrap starting.
I20260620 02:57:22.995155 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:57:22.996004 21094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dad06b2526274022acaa8af702c05b54: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:22.997119 21094 log.cc:824] T 00000000000000000000000000000000 P dad06b2526274022acaa8af702c05b54: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:23.001143 21094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dad06b2526274022acaa8af702c05b54: No bootstrap required, opened a new log
I20260620 02:57:23.002812 21097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:23.000049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56463 (local address 127.0.0.1:35755)
0620 02:57:23.000221 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:23.000225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:23.000753 (+ 528us) server_negotiation.cc:407] Connection header received
0620 02:57:23.000964 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:23.000969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:23.001032 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:23.001148 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:23.002126 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:23.002132 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:23.002137 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:23.002155 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:23.002203 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:23.002218 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:23.002345 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:23.002474 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 02:57:23.002534 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 02:57:23.013574 21094 raft_consensus.cc:348] T 00000000000000000000000000000000 P dad06b2526274022acaa8af702c05b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad06b2526274022acaa8af702c05b54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35755 } }
I20260620 02:57:23.014174 21094 raft_consensus.cc:374] T 00000000000000000000000000000000 P dad06b2526274022acaa8af702c05b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:23.014343 21094 raft_consensus.cc:729] T 00000000000000000000000000000000 P dad06b2526274022acaa8af702c05b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dad06b2526274022acaa8af702c05b54, State: Initialized, Role: FOLLOWER
I20260620 02:57:23.015362 21094 consensus_queue.cc:260] T 00000000000000000000000000000000 P dad06b2526274022acaa8af702c05b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad06b2526274022acaa8af702c05b54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35755 } }
I20260620 02:57:23.015678 21094 raft_consensus.cc:388] T 00000000000000000000000000000000 P dad06b2526274022acaa8af702c05b54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:57:23.015829 21094 raft_consensus.cc:482] T 00000000000000000000000000000000 P dad06b2526274022acaa8af702c05b54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:57:23.016027 21094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dad06b2526274022acaa8af702c05b54 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:23.020267 21094 raft_consensus.cc:504] T 00000000000000000000000000000000 P dad06b2526274022acaa8af702c05b54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad06b2526274022acaa8af702c05b54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35755 } }
I20260620 02:57:23.021212 21094 leader_election.cc:302] T 00000000000000000000000000000000 P dad06b2526274022acaa8af702c05b54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dad06b2526274022acaa8af702c05b54; no voters:
I20260620 02:57:23.022043 21094 leader_election.cc:288] T 00000000000000000000000000000000 P dad06b2526274022acaa8af702c05b54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:57:23.022161 21104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dad06b2526274022acaa8af702c05b54 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:57:23.023535 21104 raft_consensus.cc:686] T 00000000000000000000000000000000 P dad06b2526274022acaa8af702c05b54 [term 1 LEADER]: Becoming Leader. State: Replica: dad06b2526274022acaa8af702c05b54, State: Running, Role: LEADER
I20260620 02:57:23.024194 21104 consensus_queue.cc:237] T 00000000000000000000000000000000 P dad06b2526274022acaa8af702c05b54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad06b2526274022acaa8af702c05b54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35755 } }
I20260620 02:57:23.024688 21094 sys_catalog.cc:564] T 00000000000000000000000000000000 P dad06b2526274022acaa8af702c05b54 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:57:23.027748 21106 sys_catalog.cc:455] T 00000000000000000000000000000000 P dad06b2526274022acaa8af702c05b54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dad06b2526274022acaa8af702c05b54. Latest consensus state: current_term: 1 leader_uuid: "dad06b2526274022acaa8af702c05b54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad06b2526274022acaa8af702c05b54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35755 } } }
I20260620 02:57:23.028654 21106 sys_catalog.cc:458] T 00000000000000000000000000000000 P dad06b2526274022acaa8af702c05b54 [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:23.028165 21105 sys_catalog.cc:455] T 00000000000000000000000000000000 P dad06b2526274022acaa8af702c05b54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dad06b2526274022acaa8af702c05b54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad06b2526274022acaa8af702c05b54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35755 } } }
I20260620 02:57:23.029695 21105 sys_catalog.cc:458] T 00000000000000000000000000000000 P dad06b2526274022acaa8af702c05b54 [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:23.040738 21114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:57:23.047118 21114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:57:23.058239 21114 catalog_manager.cc:1347] Generated new cluster ID: 5b655fcbf14847c18ff872d8600081bd
I20260620 02:57:23.058413 21114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:57:23.095638 21114 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:57:23.096186 21114 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:57:23.104058 21114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dad06b2526274022acaa8af702c05b54: Generated new TSK 0
I20260620 02:57:23.104877 21114 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:57:23.119902 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 21027
W20260620 02:57:23.121191 21089 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49179
W20260620 02:57:23.121948 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 21027: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:57:23.122097 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 21027
2026-06-20T02:57:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1958 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:57:23 dist-test-slave-8p8c krb5kdc[21125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:57:23 dist-test-slave-8p8c krb5kdc[21125](info): set up 2 sockets
Jun 20 02:57:23 dist-test-slave-8p8c krb5kdc[21125](info): commencing operation
krb5kdc: starting...
W20260620 02:57:24.230808 388 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:57:24 dist-test-slave-8p8c krb5kdc[21125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:57:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:57:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:57:24.493407 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46191
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:57:24.821022 21138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:24.821334 21138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:24.821499 21138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:24.828862 21138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:57:24.828980 21138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:24.829100 21138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:24.829188 21138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:57:24.829257 21138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:57:24.839489 21138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:24.840888 21138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:24.842264 21138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:24.854218 21146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:24.854303 21143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:24.854278 21144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:24.855800 21138 server_base.cc:1034] running on GCE node
I20260620 02:57:24.856858 21138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:24.858644 21138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:24.859913 21138 hybrid_clock.cc:648] HybridClock initialized: now 1781924244859853 us; error 52 us; skew 500 ppm
I20260620 02:57:24.860267 21138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:57:24 dist-test-slave-8p8c krb5kdc[21125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:57:24.867755 21138 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:57:24.869397 21138 webserver.cc:460] Webserver started at http://127.0.0.1:46593/ using document root <none> and password file <none>
I20260620 02:57:24.870095 21138 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:24.870203 21138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:24.870553 21138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:24.873557 21138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "7c4d5f1cae124bde8c72484e2c4c4170"
format_stamp: "Formatted at 2026-06-20 02:57:24 on dist-test-slave-8p8c"
server_key: "9b9804d04f24bf1eb7f14da15d4b3bc5"
server_key_iv: "eb4056be5429cadc4dc30a5f5dfc65dc"
server_key_version: "encryptionkey@0"
I20260620 02:57:24.874293 21138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "7c4d5f1cae124bde8c72484e2c4c4170"
format_stamp: "Formatted at 2026-06-20 02:57:24 on dist-test-slave-8p8c"
server_key: "9b9804d04f24bf1eb7f14da15d4b3bc5"
server_key_iv: "eb4056be5429cadc4dc30a5f5dfc65dc"
server_key_version: "encryptionkey@0"
I20260620 02:57:24.879928 21138 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 02:57:24.883728 21153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:24.885265 21138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:57:24.885422 21138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "7c4d5f1cae124bde8c72484e2c4c4170"
format_stamp: "Formatted at 2026-06-20 02:57:24 on dist-test-slave-8p8c"
server_key: "9b9804d04f24bf1eb7f14da15d4b3bc5"
server_key_iv: "eb4056be5429cadc4dc30a5f5dfc65dc"
server_key_version: "encryptionkey@0"
I20260620 02:57:24.885635 21138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:24.917783 21138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:24.922195 21138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:24.922508 21138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:24.941768 21138 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46191
I20260620 02:57:24.941803 21204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46191 every 8 connection(s)
I20260620 02:57:24.943259 21138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:57:24.947230 21205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:24.947892 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 21138
I20260620 02:57:24.948081 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:57:24.948654 388 external_mini_cluster.cc:1428] Setting key b1b22efa650e95349ddb678b776111ef
I20260620 02:57:24.961797 21205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c4d5f1cae124bde8c72484e2c4c4170: Bootstrap starting.
Jun 20 02:57:24 dist-test-slave-8p8c krb5kdc[21125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 02:57:24.965862 21205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c4d5f1cae124bde8c72484e2c4c4170: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:24.967041 21205 log.cc:824] T 00000000000000000000000000000000 P 7c4d5f1cae124bde8c72484e2c4c4170: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:24.969895 21205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c4d5f1cae124bde8c72484e2c4c4170: No bootstrap required, opened a new log
I20260620 02:57:24.972815 21208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:24.950550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35781 (local address 127.0.0.1:46191)
0620 02:57:24.951354 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:24.951365 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:24.951406 (+ 41us) server_negotiation.cc:407] Connection header received
0620 02:57:24.953576 (+ 2170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:24.953601 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:24.954043 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:24.954542 (+ 499us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:24.956499 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:24.957860 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:24.959809 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:24.960501 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:24.960655 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:24.965427 (+ 4772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:24.965436 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:24.965451 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:24.965494 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:24.968623 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:24.969340 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:24.969347 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:24.969353 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:24.969523 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:24.969925 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:24.969929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:24.969932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:24.970587 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:24.970893 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:24.971342 (+ 449us) server_negotiation.cc:299] Negotiation successful
0620 02:57:24.971797 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":391,"threads_started":1}
I20260620 02:57:24.976555 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:57:24.978224 21205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c4d5f1cae124bde8c72484e2c4c4170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c4d5f1cae124bde8c72484e2c4c4170" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46191 } }
I20260620 02:57:24.978653 21205 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c4d5f1cae124bde8c72484e2c4c4170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:24.978758 21205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c4d5f1cae124bde8c72484e2c4c4170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c4d5f1cae124bde8c72484e2c4c4170, State: Initialized, Role: FOLLOWER
I20260620 02:57:24.979403 21205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c4d5f1cae124bde8c72484e2c4c4170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c4d5f1cae124bde8c72484e2c4c4170" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46191 } }
I20260620 02:57:24.979676 21205 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c4d5f1cae124bde8c72484e2c4c4170 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:57:24.979807 21205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c4d5f1cae124bde8c72484e2c4c4170 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:57:24.979990 21205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c4d5f1cae124bde8c72484e2c4c4170 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:24.982919 21205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c4d5f1cae124bde8c72484e2c4c4170 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c4d5f1cae124bde8c72484e2c4c4170" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46191 } }
I20260620 02:57:24.983511 21205 leader_election.cc:302] T 00000000000000000000000000000000 P 7c4d5f1cae124bde8c72484e2c4c4170 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c4d5f1cae124bde8c72484e2c4c4170; no voters:
I20260620 02:57:24.984104 21205 leader_election.cc:288] T 00000000000000000000000000000000 P 7c4d5f1cae124bde8c72484e2c4c4170 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:57:24.984263 21215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c4d5f1cae124bde8c72484e2c4c4170 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:57:24.986243 21205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c4d5f1cae124bde8c72484e2c4c4170 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:57:24.991254 21215 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c4d5f1cae124bde8c72484e2c4c4170 [term 1 LEADER]: Becoming Leader. State: Replica: 7c4d5f1cae124bde8c72484e2c4c4170, State: Running, Role: LEADER
I20260620 02:57:24.991935 21215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c4d5f1cae124bde8c72484e2c4c4170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c4d5f1cae124bde8c72484e2c4c4170" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46191 } }
I20260620 02:57:24.994881 21216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c4d5f1cae124bde8c72484e2c4c4170 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c4d5f1cae124bde8c72484e2c4c4170" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c4d5f1cae124bde8c72484e2c4c4170" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46191 } } }
I20260620 02:57:24.995368 21216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c4d5f1cae124bde8c72484e2c4c4170 [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:24.995299 21217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c4d5f1cae124bde8c72484e2c4c4170 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c4d5f1cae124bde8c72484e2c4c4170. Latest consensus state: current_term: 1 leader_uuid: "7c4d5f1cae124bde8c72484e2c4c4170" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c4d5f1cae124bde8c72484e2c4c4170" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46191 } } }
I20260620 02:57:24.996417 21217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c4d5f1cae124bde8c72484e2c4c4170 [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:24.997270 21222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:57:25.003736 21208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:24.981165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46343 (local address 127.0.0.1:46191)
0620 02:57:24.981355 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:24.981361 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:24.981884 (+ 523us) server_negotiation.cc:407] Connection header received
0620 02:57:24.982001 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:24.982006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:24.982082 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:24.982187 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:24.984205 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:24.985384 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:24.987782 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:24.988865 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:24.989029 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:24.992645 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:24.992655 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:24.992663 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:24.992721 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:24.997963 (+ 5242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:24.999363 (+ 1400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:24.999372 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:24.999378 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:24.999592 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:25.000181 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:25.000189 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:25.000195 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:25.000814 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:25.002644 (+ 1830us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:25.002839 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 02:57:25.003161 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 02:57:25.013129 21222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:57:25.022948 21222 catalog_manager.cc:1347] Generated new cluster ID: 328c47ec1e974d93a728745b41c59a55
I20260620 02:57:25.023064 21222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:57:25.047333 21222 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:57:25.048781 21222 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:57:25.055570 21222 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c4d5f1cae124bde8c72484e2c4c4170: Generated new TSK 0
I20260620 02:57:25.056406 21222 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:57:25.062639 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 21138
I20260620 02:57:25.064374 21200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:57:25.156368 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 21138
2026-06-20T02:57:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2036 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:57:25 dist-test-slave-8p8c krb5kdc[21237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:57:25 dist-test-slave-8p8c krb5kdc[21237](info): set up 2 sockets
Jun 20 02:57:25 dist-test-slave-8p8c krb5kdc[21237](info): commencing operation
krb5kdc: starting...
W20260620 02:57:26.266952 388 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:57:26 dist-test-slave-8p8c krb5kdc[21237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:57:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:57:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:57:26.538790 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41057
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:57:26.872185 21250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:26.872499 21250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:26.872670 21250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:26.880164 21250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:57:26.880285 21250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:26.880366 21250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:26.880446 21250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:57:26.880542 21250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:57:26.891038 21250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:26.892408 21250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:26.893797 21250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:26.906239 21255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:26.906127 21258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:26.906203 21256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:26.907683 21250 server_base.cc:1034] running on GCE node
I20260620 02:57:26.908485 21250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:26.909945 21250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:26.911191 21250 hybrid_clock.cc:648] HybridClock initialized: now 1781924246911131 us; error 61 us; skew 500 ppm
I20260620 02:57:26.911741 21250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:57:26 dist-test-slave-8p8c krb5kdc[21237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:57:26.915560 21250 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:57:26.917214 21250 webserver.cc:460] Webserver started at http://127.0.0.1:46681/ using document root <none> and password file <none>
I20260620 02:57:26.917945 21250 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:26.918103 21250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:26.918480 21250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:26.922348 21250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "b6e11ea0c5f844748950311fe70bd800"
format_stamp: "Formatted at 2026-06-20 02:57:26 on dist-test-slave-8p8c"
server_key: "2dfb38665c0de1481a494606697e2f04"
server_key_iv: "57d930b3688f65a9a3dad18c10ca2f32"
server_key_version: "encryptionkey@0"
I20260620 02:57:26.923334 21250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "b6e11ea0c5f844748950311fe70bd800"
format_stamp: "Formatted at 2026-06-20 02:57:26 on dist-test-slave-8p8c"
server_key: "2dfb38665c0de1481a494606697e2f04"
server_key_iv: "57d930b3688f65a9a3dad18c10ca2f32"
server_key_version: "encryptionkey@0"
I20260620 02:57:26.928913 21250 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260620 02:57:26.932762 21265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:26.934252 21250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:57:26.934487 21250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "b6e11ea0c5f844748950311fe70bd800"
format_stamp: "Formatted at 2026-06-20 02:57:26 on dist-test-slave-8p8c"
server_key: "2dfb38665c0de1481a494606697e2f04"
server_key_iv: "57d930b3688f65a9a3dad18c10ca2f32"
server_key_version: "encryptionkey@0"
I20260620 02:57:26.934737 21250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:26.965130 21250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:26.970734 21250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:26.971048 21250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:26.992053 21250 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41057
I20260620 02:57:26.992025 21316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41057 every 8 connection(s)
I20260620 02:57:26.993512 21250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:57:26.994834 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 21250
I20260620 02:57:26.995038 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:57:26.995584 388 external_mini_cluster.cc:1428] Setting key 07d1124c7627cb6230636c2c4354052e
I20260620 02:57:26.998684 21317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:27.011976 21320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:26.997706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54855 (local address 127.0.0.1:41057)
0620 02:57:26.998566 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:26.998579 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:26.998630 (+ 51us) server_negotiation.cc:407] Connection header received
0620 02:57:27.000345 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:27.000370 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:27.001014 (+ 644us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:27.001395 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:27.002052 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:27.004188 (+ 2136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:27.006127 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:27.006951 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:27.007259 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:27.008837 (+ 1578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:27.008849 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:27.008870 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:27.009687 (+ 817us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:27.009790 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:27.009837 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:27.009947 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:27.010222 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 02:57:27.010571 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":389,"threads_started":1}
I20260620 02:57:27.013396 21317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6e11ea0c5f844748950311fe70bd800: Bootstrap starting.
I20260620 02:57:27.015733 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:57:27.016901 21317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6e11ea0c5f844748950311fe70bd800: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:27.018095 21317 log.cc:824] T 00000000000000000000000000000000 P b6e11ea0c5f844748950311fe70bd800: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:27.021557 21317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6e11ea0c5f844748950311fe70bd800: No bootstrap required, opened a new log
I20260620 02:57:27.028036 21320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:27.021089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41727 (local address 127.0.0.1:41057)
0620 02:57:27.021253 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:27.021257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:27.021810 (+ 553us) server_negotiation.cc:407] Connection header received
0620 02:57:27.022041 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:27.022049 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:27.022145 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:27.022257 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:27.022832 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:27.023954 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:27.025335 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:27.026026 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:27.026153 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:27.027307 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:27.027314 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:27.027319 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:27.027345 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:27.027389 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:27.027408 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:27.027506 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:27.027592 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 02:57:27.027687 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 02:57:27.031364 21317 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6e11ea0c5f844748950311fe70bd800 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e11ea0c5f844748950311fe70bd800" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41057 } }
I20260620 02:57:27.031733 21317 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6e11ea0c5f844748950311fe70bd800 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:27.031862 21317 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6e11ea0c5f844748950311fe70bd800 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6e11ea0c5f844748950311fe70bd800, State: Initialized, Role: FOLLOWER
I20260620 02:57:27.032457 21317 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6e11ea0c5f844748950311fe70bd800 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e11ea0c5f844748950311fe70bd800" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41057 } }
I20260620 02:57:27.032685 21317 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6e11ea0c5f844748950311fe70bd800 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:57:27.032864 21317 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6e11ea0c5f844748950311fe70bd800 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:57:27.033025 21317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6e11ea0c5f844748950311fe70bd800 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:27.035490 21317 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6e11ea0c5f844748950311fe70bd800 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e11ea0c5f844748950311fe70bd800" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41057 } }
I20260620 02:57:27.036041 21317 leader_election.cc:302] T 00000000000000000000000000000000 P b6e11ea0c5f844748950311fe70bd800 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6e11ea0c5f844748950311fe70bd800; no voters:
I20260620 02:57:27.036767 21317 leader_election.cc:288] T 00000000000000000000000000000000 P b6e11ea0c5f844748950311fe70bd800 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:57:27.037014 21327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6e11ea0c5f844748950311fe70bd800 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:57:27.037761 21327 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6e11ea0c5f844748950311fe70bd800 [term 1 LEADER]: Becoming Leader. State: Replica: b6e11ea0c5f844748950311fe70bd800, State: Running, Role: LEADER
I20260620 02:57:27.038295 21327 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6e11ea0c5f844748950311fe70bd800 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e11ea0c5f844748950311fe70bd800" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41057 } }
I20260620 02:57:27.039037 21317 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6e11ea0c5f844748950311fe70bd800 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:57:27.042126 21328 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6e11ea0c5f844748950311fe70bd800 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6e11ea0c5f844748950311fe70bd800" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e11ea0c5f844748950311fe70bd800" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41057 } } }
I20260620 02:57:27.042693 21328 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6e11ea0c5f844748950311fe70bd800 [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:27.042512 21329 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6e11ea0c5f844748950311fe70bd800 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6e11ea0c5f844748950311fe70bd800. Latest consensus state: current_term: 1 leader_uuid: "b6e11ea0c5f844748950311fe70bd800" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e11ea0c5f844748950311fe70bd800" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41057 } } }
I20260620 02:57:27.042896 21329 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6e11ea0c5f844748950311fe70bd800 [sys.catalog]: This master's current role is: LEADER
W20260620 02:57:27.051528 21343 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b6e11ea0c5f844748950311fe70bd800: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:57:27.051685 21343 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:57:27.052356 21336 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:57:27.054261 21336 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:57:27.063218 21336 catalog_manager.cc:1347] Generated new cluster ID: f983ae1ef2a44361a44358dc31d4fe44
I20260620 02:57:27.063371 21336 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:57:27.093772 21336 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:57:27.095520 21336 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:57:27.103778 21336 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6e11ea0c5f844748950311fe70bd800: Generated new TSK 0
I20260620 02:57:27.104655 21336 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:57:27.114847 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 21250
W20260620 02:57:27.116016 21312 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54855
W20260620 02:57:27.116829 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 21250: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:57:27.117043 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 21250
2026-06-20T02:57:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1959 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:57:27 dist-test-slave-8p8c krb5kdc[21348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:57:27 dist-test-slave-8p8c krb5kdc[21348](info): set up 2 sockets
Jun 20 02:57:27 dist-test-slave-8p8c krb5kdc[21348](info): commencing operation
krb5kdc: starting...
W20260620 02:57:28.223297 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:57:28 dist-test-slave-8p8c krb5kdc[21348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:57:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:57:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:57:28.494415 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34203
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:57:28.826855 21361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:28.827275 21361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:28.827378 21361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:28.834594 21361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:57:28.834707 21361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:28.834753 21361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:28.834806 21361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:57:28.834851 21361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:57:28.844888 21361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:28.846271 21361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:28.847751 21361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:28.859696 21366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:28.859746 21367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:28.859758 21369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:28.860306 21361 server_base.cc:1034] running on GCE node
I20260620 02:57:28.861294 21361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:28.862759 21361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:28.863961 21361 hybrid_clock.cc:648] HybridClock initialized: now 1781924248863900 us; error 52 us; skew 500 ppm
I20260620 02:57:28.864279 21361 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:57:28 dist-test-slave-8p8c krb5kdc[21348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:57:28.867565 21361 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:57:28.869078 21361 webserver.cc:460] Webserver started at http://127.0.0.1:33907/ using document root <none> and password file <none>
I20260620 02:57:28.869724 21361 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:28.869822 21361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:28.870090 21361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:28.873708 21361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "dead7a00b13b4e10841a6d42a2cde832"
format_stamp: "Formatted at 2026-06-20 02:57:28 on dist-test-slave-8p8c"
server_key: "28f27bb2a618f1e64b3dc213c25d8f00"
server_key_iv: "17aaf659e4edef47a6d90434c679351e"
server_key_version: "encryptionkey@0"
I20260620 02:57:28.874703 21361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "dead7a00b13b4e10841a6d42a2cde832"
format_stamp: "Formatted at 2026-06-20 02:57:28 on dist-test-slave-8p8c"
server_key: "28f27bb2a618f1e64b3dc213c25d8f00"
server_key_iv: "17aaf659e4edef47a6d90434c679351e"
server_key_version: "encryptionkey@0"
I20260620 02:57:28.880383 21361 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 02:57:28.883976 21376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:28.885423 21361 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:57:28.885576 21361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "dead7a00b13b4e10841a6d42a2cde832"
format_stamp: "Formatted at 2026-06-20 02:57:28 on dist-test-slave-8p8c"
server_key: "28f27bb2a618f1e64b3dc213c25d8f00"
server_key_iv: "17aaf659e4edef47a6d90434c679351e"
server_key_version: "encryptionkey@0"
I20260620 02:57:28.885753 21361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:28.891633 21361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:28.896510 21361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:28.896764 21361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:28.916260 21361 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34203
I20260620 02:57:28.916370 21427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34203 every 8 connection(s)
I20260620 02:57:28.917745 21361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:57:28.918982 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 21361
I20260620 02:57:28.919201 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:57:28.919629 388 external_mini_cluster.cc:1428] Setting key 02d851988c32dbcc6117e839e877a52a
I20260620 02:57:28.922941 21428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:28.927778 21431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:28.921578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45121 (local address 127.0.0.1:34203)
0620 02:57:28.922511 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:28.922524 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:28.922568 (+ 44us) server_negotiation.cc:407] Connection header received
0620 02:57:28.924311 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:28.924335 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:28.924680 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:28.925077 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:28.926149 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:28.926155 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:28.926167 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:28.926180 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:28.926264 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:28.926283 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:28.926549 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:28.926700 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 02:57:28.926882 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":364,"threads_started":1}
I20260620 02:57:28.931545 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:57:28.935281 21428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dead7a00b13b4e10841a6d42a2cde832: Bootstrap starting.
I20260620 02:57:28.938905 21431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:28.935899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40061 (local address 127.0.0.1:34203)
0620 02:57:28.936086 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:28.936092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:28.936600 (+ 508us) server_negotiation.cc:407] Connection header received
0620 02:57:28.936740 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:28.936745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:28.936815 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:28.936951 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:28.937919 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:28.937925 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:28.937929 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:28.937951 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:28.938002 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:28.938023 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:28.938143 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:28.938518 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 02:57:28.938587 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 02:57:28.939589 21428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dead7a00b13b4e10841a6d42a2cde832: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:28.940724 21428 log.cc:824] T 00000000000000000000000000000000 P dead7a00b13b4e10841a6d42a2cde832: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:28.943827 21428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dead7a00b13b4e10841a6d42a2cde832: No bootstrap required, opened a new log
I20260620 02:57:28.951095 21428 raft_consensus.cc:348] T 00000000000000000000000000000000 P dead7a00b13b4e10841a6d42a2cde832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dead7a00b13b4e10841a6d42a2cde832" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34203 } }
I20260620 02:57:28.951526 21428 raft_consensus.cc:374] T 00000000000000000000000000000000 P dead7a00b13b4e10841a6d42a2cde832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:28.951637 21428 raft_consensus.cc:729] T 00000000000000000000000000000000 P dead7a00b13b4e10841a6d42a2cde832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dead7a00b13b4e10841a6d42a2cde832, State: Initialized, Role: FOLLOWER
I20260620 02:57:28.952219 21428 consensus_queue.cc:260] T 00000000000000000000000000000000 P dead7a00b13b4e10841a6d42a2cde832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dead7a00b13b4e10841a6d42a2cde832" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34203 } }
I20260620 02:57:28.952471 21428 raft_consensus.cc:388] T 00000000000000000000000000000000 P dead7a00b13b4e10841a6d42a2cde832 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:57:28.952656 21428 raft_consensus.cc:482] T 00000000000000000000000000000000 P dead7a00b13b4e10841a6d42a2cde832 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:57:28.952886 21428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dead7a00b13b4e10841a6d42a2cde832 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:28.955176 21428 raft_consensus.cc:504] T 00000000000000000000000000000000 P dead7a00b13b4e10841a6d42a2cde832 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dead7a00b13b4e10841a6d42a2cde832" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34203 } }
I20260620 02:57:28.955703 21428 leader_election.cc:302] T 00000000000000000000000000000000 P dead7a00b13b4e10841a6d42a2cde832 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dead7a00b13b4e10841a6d42a2cde832; no voters:
I20260620 02:57:28.956331 21428 leader_election.cc:288] T 00000000000000000000000000000000 P dead7a00b13b4e10841a6d42a2cde832 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:57:28.956497 21438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dead7a00b13b4e10841a6d42a2cde832 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:57:28.957172 21438 raft_consensus.cc:686] T 00000000000000000000000000000000 P dead7a00b13b4e10841a6d42a2cde832 [term 1 LEADER]: Becoming Leader. State: Replica: dead7a00b13b4e10841a6d42a2cde832, State: Running, Role: LEADER
I20260620 02:57:28.957669 21438 consensus_queue.cc:237] T 00000000000000000000000000000000 P dead7a00b13b4e10841a6d42a2cde832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dead7a00b13b4e10841a6d42a2cde832" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34203 } }
I20260620 02:57:28.958489 21428 sys_catalog.cc:564] T 00000000000000000000000000000000 P dead7a00b13b4e10841a6d42a2cde832 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:57:28.960805 21439 sys_catalog.cc:455] T 00000000000000000000000000000000 P dead7a00b13b4e10841a6d42a2cde832 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dead7a00b13b4e10841a6d42a2cde832" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dead7a00b13b4e10841a6d42a2cde832" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34203 } } }
I20260620 02:57:28.960901 21440 sys_catalog.cc:455] T 00000000000000000000000000000000 P dead7a00b13b4e10841a6d42a2cde832 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dead7a00b13b4e10841a6d42a2cde832. Latest consensus state: current_term: 1 leader_uuid: "dead7a00b13b4e10841a6d42a2cde832" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dead7a00b13b4e10841a6d42a2cde832" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34203 } } }
I20260620 02:57:28.961267 21439 sys_catalog.cc:458] T 00000000000000000000000000000000 P dead7a00b13b4e10841a6d42a2cde832 [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:28.961337 21440 sys_catalog.cc:458] T 00000000000000000000000000000000 P dead7a00b13b4e10841a6d42a2cde832 [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:28.962354 21445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:57:28.968667 21445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:57:28.978729 21445 catalog_manager.cc:1347] Generated new cluster ID: d1953d2dfd2f4ffaa77758a5b1a91cd4
I20260620 02:57:28.978873 21445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:57:28.992324 21445 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:57:28.992771 21445 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:57:29.009954 21445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dead7a00b13b4e10841a6d42a2cde832: Generated new TSK 0
I20260620 02:57:29.010921 21445 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:57:29.017690 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 21361
W20260620 02:57:29.019385 21423 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45121
W20260620 02:57:29.020262 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 21361: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:57:29.020447 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 21361
2026-06-20T02:57:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1904 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:57:29 dist-test-slave-8p8c krb5kdc[21459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:57:29 dist-test-slave-8p8c krb5kdc[21459](info): set up 2 sockets
Jun 20 02:57:29 dist-test-slave-8p8c krb5kdc[21459](info): commencing operation
krb5kdc: starting...
W20260620 02:57:30.126796 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:57:30 dist-test-slave-8p8c krb5kdc[21459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:57:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:57:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:57:30.377260 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39765
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:57:30.712845 21472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:30.713156 21472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:30.713254 21472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:30.720568 21472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:57:30.720685 21472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:30.720731 21472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:30.720785 21472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:57:30.720829 21472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:57:30.730701 21472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:30.731948 21472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:30.733273 21472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:30.745204 21477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:30.745195 21480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:30.745338 21478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:30.745819 21472 server_base.cc:1034] running on GCE node
I20260620 02:57:30.747031 21472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:30.748867 21472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:30.750144 21472 hybrid_clock.cc:648] HybridClock initialized: now 1781924250750094 us; error 41 us; skew 500 ppm
I20260620 02:57:30.750622 21472 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:57:30 dist-test-slave-8p8c krb5kdc[21459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:57:30.759548 21472 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:57:30.761304 21472 webserver.cc:460] Webserver started at http://127.0.0.1:38809/ using document root <none> and password file <none>
I20260620 02:57:30.762079 21472 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:30.762239 21472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:30.762619 21472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:30.765808 21472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "375bf2f3ff8441d5932572c2205448e6"
format_stamp: "Formatted at 2026-06-20 02:57:30 on dist-test-slave-8p8c"
server_key: "8a46d56e5523dcfcb7f895504fb98122"
server_key_iv: "459357c1bbdf759103e210c2ad50aac3"
server_key_version: "encryptionkey@0"
I20260620 02:57:30.766696 21472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "375bf2f3ff8441d5932572c2205448e6"
format_stamp: "Formatted at 2026-06-20 02:57:30 on dist-test-slave-8p8c"
server_key: "8a46d56e5523dcfcb7f895504fb98122"
server_key_iv: "459357c1bbdf759103e210c2ad50aac3"
server_key_version: "encryptionkey@0"
I20260620 02:57:30.772156 21472 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:57:30.775750 21487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:30.777264 21472 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:57:30.777473 21472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "375bf2f3ff8441d5932572c2205448e6"
format_stamp: "Formatted at 2026-06-20 02:57:30 on dist-test-slave-8p8c"
server_key: "8a46d56e5523dcfcb7f895504fb98122"
server_key_iv: "459357c1bbdf759103e210c2ad50aac3"
server_key_version: "encryptionkey@0"
I20260620 02:57:30.777684 21472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:30.795326 21472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:30.800572 21472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:30.800896 21472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:30.820465 21472 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39765
I20260620 02:57:30.820516 21538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39765 every 8 connection(s)
I20260620 02:57:30.821897 21472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:57:30.822489 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 21472
I20260620 02:57:30.822708 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:57:30.823316 388 external_mini_cluster.cc:1428] Setting key a06cff447f09f6d69dd2bf7a6593ab08
I20260620 02:57:30.826821 21539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 02:57:30 dist-test-slave-8p8c krb5kdc[21459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 02:57:30.838160 21539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 375bf2f3ff8441d5932572c2205448e6: Bootstrap starting.
I20260620 02:57:30.841735 21539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 375bf2f3ff8441d5932572c2205448e6: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:30.842907 21539 log.cc:824] T 00000000000000000000000000000000 P 375bf2f3ff8441d5932572c2205448e6: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:30.845988 21539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 375bf2f3ff8441d5932572c2205448e6: No bootstrap required, opened a new log
I20260620 02:57:30.846951 21542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:30.825210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.166:54519 (local address 127.0.0.1:39765)
0620 02:57:30.825964 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:30.825978 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:30.826019 (+ 41us) server_negotiation.cc:407] Connection header received
0620 02:57:30.827787 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:30.827810 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:30.828212 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:30.828592 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:30.830050 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:30.831816 (+ 1766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:30.833127 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:30.833809 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:30.834017 (+ 208us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:30.838604 (+ 4587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:30.838613 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:30.838629 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:30.838675 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:30.842212 (+ 3537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:30.843115 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:30.843122 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:30.843129 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:30.843295 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:30.843770 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:30.843781 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:30.843784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:30.844433 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:30.844788 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:30.845342 (+ 554us) server_negotiation.cc:299] Negotiation successful
0620 02:57:30.845672 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":339,"threads_started":1}
I20260620 02:57:30.851107 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:57:30.854522 21539 raft_consensus.cc:348] T 00000000000000000000000000000000 P 375bf2f3ff8441d5932572c2205448e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375bf2f3ff8441d5932572c2205448e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39765 } }
I20260620 02:57:30.854928 21539 raft_consensus.cc:374] T 00000000000000000000000000000000 P 375bf2f3ff8441d5932572c2205448e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:30.855032 21539 raft_consensus.cc:729] T 00000000000000000000000000000000 P 375bf2f3ff8441d5932572c2205448e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 375bf2f3ff8441d5932572c2205448e6, State: Initialized, Role: FOLLOWER
I20260620 02:57:30.855612 21539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 375bf2f3ff8441d5932572c2205448e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375bf2f3ff8441d5932572c2205448e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39765 } }
I20260620 02:57:30.855933 21539 raft_consensus.cc:388] T 00000000000000000000000000000000 P 375bf2f3ff8441d5932572c2205448e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:57:30.856105 21539 raft_consensus.cc:482] T 00000000000000000000000000000000 P 375bf2f3ff8441d5932572c2205448e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:57:30.856329 21539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 375bf2f3ff8441d5932572c2205448e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:30.859541 21539 raft_consensus.cc:504] T 00000000000000000000000000000000 P 375bf2f3ff8441d5932572c2205448e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375bf2f3ff8441d5932572c2205448e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39765 } }
I20260620 02:57:30.860157 21539 leader_election.cc:302] T 00000000000000000000000000000000 P 375bf2f3ff8441d5932572c2205448e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 375bf2f3ff8441d5932572c2205448e6; no voters:
I20260620 02:57:30.860806 21539 leader_election.cc:288] T 00000000000000000000000000000000 P 375bf2f3ff8441d5932572c2205448e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:57:30.860920 21549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 375bf2f3ff8441d5932572c2205448e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:57:30.861665 21549 raft_consensus.cc:686] T 00000000000000000000000000000000 P 375bf2f3ff8441d5932572c2205448e6 [term 1 LEADER]: Becoming Leader. State: Replica: 375bf2f3ff8441d5932572c2205448e6, State: Running, Role: LEADER
I20260620 02:57:30.863140 21549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 375bf2f3ff8441d5932572c2205448e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375bf2f3ff8441d5932572c2205448e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39765 } }
I20260620 02:57:30.864272 21539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 375bf2f3ff8441d5932572c2205448e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:57:30.866328 21551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 375bf2f3ff8441d5932572c2205448e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "375bf2f3ff8441d5932572c2205448e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375bf2f3ff8441d5932572c2205448e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39765 } } }
I20260620 02:57:30.866801 21551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 375bf2f3ff8441d5932572c2205448e6 [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:30.866384 21550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 375bf2f3ff8441d5932572c2205448e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 375bf2f3ff8441d5932572c2205448e6. Latest consensus state: current_term: 1 leader_uuid: "375bf2f3ff8441d5932572c2205448e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375bf2f3ff8441d5932572c2205448e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39765 } } }
I20260620 02:57:30.867375 21550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 375bf2f3ff8441d5932572c2205448e6 [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:30.876001 21542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:30.855996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.166:42609 (local address 127.0.0.1:39765)
0620 02:57:30.856251 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:30.856260 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:30.856718 (+ 458us) server_negotiation.cc:407] Connection header received
0620 02:57:30.856884 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:30.856889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:30.856957 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:30.857053 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:30.858926 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:30.859832 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:30.861855 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:30.862775 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:30.862921 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:30.866406 (+ 3485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:30.866418 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:30.866423 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:30.866474 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:30.871075 (+ 4601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:30.872377 (+ 1302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:30.872386 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:30.872393 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:30.872645 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:30.873388 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:30.873397 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:30.873403 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:30.873906 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:30.874269 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:30.875218 (+ 949us) server_negotiation.cc:299] Negotiation successful
0620 02:57:30.875492 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 02:57:30.882112 21554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:57:30.895685 21554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:57:30.904994 21554 catalog_manager.cc:1347] Generated new cluster ID: 3d38a3e7229f49b59db559218920ecde
I20260620 02:57:30.905102 21554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:57:30.915339 21554 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:57:30.916731 21554 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:57:30.935009 21554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 375bf2f3ff8441d5932572c2205448e6: Generated new TSK 0
I20260620 02:57:30.936175 21554 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:57:30.946472 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 21472
I20260620 02:57:30.947930 21534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:57:31.042199 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 21472
W20260620 02:57:31.060688 21457 connection.cc:535] client connection to 127.0.0.1:39765 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39765): Connection reset by peer (error 104)
2026-06-20T02:57:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2023 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:57:31 dist-test-slave-8p8c krb5kdc[21571](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:57:31 dist-test-slave-8p8c krb5kdc[21571](info): set up 2 sockets
Jun 20 02:57:31 dist-test-slave-8p8c krb5kdc[21571](info): commencing operation
krb5kdc: starting...
W20260620 02:57:32.182983 388 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:57:32 dist-test-slave-8p8c krb5kdc[21571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:57:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:57:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:57:32.443877 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41073
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:57:32.787386 21584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:32.787744 21584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:32.787845 21584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:32.794809 21584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:57:32.794914 21584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:32.794960 21584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:32.795013 21584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:57:32.795058 21584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:57:32.804821 21584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:32.806073 21584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:32.807403 21584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:32.820118 21589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:32.820174 21592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:32.820149 21590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:32.821059 21584 server_base.cc:1034] running on GCE node
I20260620 02:57:32.821930 21584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:32.823555 21584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:32.824815 21584 hybrid_clock.cc:648] HybridClock initialized: now 1781924252824769 us; error 42 us; skew 500 ppm
I20260620 02:57:32.825237 21584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:57:32 dist-test-slave-8p8c krb5kdc[21571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:57:32.828606 21584 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:57:32.830152 21584 webserver.cc:460] Webserver started at http://127.0.0.1:44911/ using document root <none> and password file <none>
I20260620 02:57:32.830899 21584 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:32.831051 21584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:32.831357 21584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:32.834297 21584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "a8ff365457ac4791a1e40beaea259c1a"
format_stamp: "Formatted at 2026-06-20 02:57:32 on dist-test-slave-8p8c"
server_key: "cda059dea378cdc68223d358707dc258"
server_key_iv: "9dfbe0b75a779d9eb7bb9ea6a360b5da"
server_key_version: "encryptionkey@0"
I20260620 02:57:32.835184 21584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "a8ff365457ac4791a1e40beaea259c1a"
format_stamp: "Formatted at 2026-06-20 02:57:32 on dist-test-slave-8p8c"
server_key: "cda059dea378cdc68223d358707dc258"
server_key_iv: "9dfbe0b75a779d9eb7bb9ea6a360b5da"
server_key_version: "encryptionkey@0"
I20260620 02:57:32.840494 21584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:57:32.844333 21599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:32.845849 21584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:57:32.846065 21584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "a8ff365457ac4791a1e40beaea259c1a"
format_stamp: "Formatted at 2026-06-20 02:57:32 on dist-test-slave-8p8c"
server_key: "cda059dea378cdc68223d358707dc258"
server_key_iv: "9dfbe0b75a779d9eb7bb9ea6a360b5da"
server_key_version: "encryptionkey@0"
I20260620 02:57:32.846275 21584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:32.888244 21584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:32.894230 21584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:32.894574 21584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:32.912931 21584 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41073
I20260620 02:57:32.912954 21650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41073 every 8 connection(s)
I20260620 02:57:32.914402 21584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:57:32.918468 21651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:32.920193 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 21584
I20260620 02:57:32.920389 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:57:32.920814 388 external_mini_cluster.cc:1428] Setting key e78a73f48952e7eca809f9725a57e872
I20260620 02:57:32.933283 21651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8ff365457ac4791a1e40beaea259c1a: Bootstrap starting.
I20260620 02:57:32.936548 21654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:32.922924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.166:44107 (local address 127.0.0.1:41073)
0620 02:57:32.923748 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:32.923765 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:32.923822 (+ 57us) server_negotiation.cc:407] Connection header received
0620 02:57:32.925697 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:32.925729 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:32.926321 (+ 592us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:32.926811 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:32.927317 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:32.929319 (+ 2002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:32.930867 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:32.931952 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:32.932257 (+ 305us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:32.933506 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:32.933517 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:32.933537 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:32.934305 (+ 768us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:32.934440 (+ 135us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:32.934473 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:32.934653 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:32.934919 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 02:57:32.935137 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":339,"threads_started":1}
I20260620 02:57:32.937500 21651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8ff365457ac4791a1e40beaea259c1a: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:32.938901 21651 log.cc:824] T 00000000000000000000000000000000 P a8ff365457ac4791a1e40beaea259c1a: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:32.941349 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:57:32.942586 21651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8ff365457ac4791a1e40beaea259c1a: No bootstrap required, opened a new log
I20260620 02:57:32.952988 21651 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8ff365457ac4791a1e40beaea259c1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ff365457ac4791a1e40beaea259c1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41073 } }
I20260620 02:57:32.953879 21651 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8ff365457ac4791a1e40beaea259c1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:32.953925 21654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:32.946059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.166:58031 (local address 127.0.0.1:41073)
0620 02:57:32.946266 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:32.946274 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:32.946778 (+ 504us) server_negotiation.cc:407] Connection header received
0620 02:57:32.947046 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:32.947050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:32.947124 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:32.947231 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:32.947786 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:32.948706 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:32.950295 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:32.950946 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:32.951088 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:32.952850 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:32.952855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:32.952858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:32.952879 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:32.952939 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:32.952956 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:32.953087 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:32.953284 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 02:57:32.953342 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 02:57:32.954015 21651 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8ff365457ac4791a1e40beaea259c1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8ff365457ac4791a1e40beaea259c1a, State: Initialized, Role: FOLLOWER
I20260620 02:57:32.954849 21651 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8ff365457ac4791a1e40beaea259c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ff365457ac4791a1e40beaea259c1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41073 } }
I20260620 02:57:32.955117 21651 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8ff365457ac4791a1e40beaea259c1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:57:32.955294 21651 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8ff365457ac4791a1e40beaea259c1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:57:32.955494 21651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8ff365457ac4791a1e40beaea259c1a [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:32.957861 21651 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8ff365457ac4791a1e40beaea259c1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ff365457ac4791a1e40beaea259c1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41073 } }
I20260620 02:57:32.958529 21651 leader_election.cc:302] T 00000000000000000000000000000000 P a8ff365457ac4791a1e40beaea259c1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8ff365457ac4791a1e40beaea259c1a; no voters:
I20260620 02:57:32.959175 21651 leader_election.cc:288] T 00000000000000000000000000000000 P a8ff365457ac4791a1e40beaea259c1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:57:32.959353 21661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8ff365457ac4791a1e40beaea259c1a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:57:32.960155 21661 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8ff365457ac4791a1e40beaea259c1a [term 1 LEADER]: Becoming Leader. State: Replica: a8ff365457ac4791a1e40beaea259c1a, State: Running, Role: LEADER
I20260620 02:57:32.960812 21661 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8ff365457ac4791a1e40beaea259c1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ff365457ac4791a1e40beaea259c1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41073 } }
I20260620 02:57:32.961671 21651 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8ff365457ac4791a1e40beaea259c1a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:57:32.964135 21663 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8ff365457ac4791a1e40beaea259c1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8ff365457ac4791a1e40beaea259c1a. Latest consensus state: current_term: 1 leader_uuid: "a8ff365457ac4791a1e40beaea259c1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ff365457ac4791a1e40beaea259c1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41073 } } }
I20260620 02:57:32.964110 21662 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8ff365457ac4791a1e40beaea259c1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8ff365457ac4791a1e40beaea259c1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ff365457ac4791a1e40beaea259c1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41073 } } }
I20260620 02:57:32.964542 21663 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8ff365457ac4791a1e40beaea259c1a [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:32.964578 21662 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8ff365457ac4791a1e40beaea259c1a [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:32.969542 21666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:57:32.976604 21666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:57:32.986722 21666 catalog_manager.cc:1347] Generated new cluster ID: 6cd092a46ae9487caa2ec24cb1cf4303
I20260620 02:57:32.986852 21666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:57:32.998276 21666 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:57:32.999681 21666 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:57:33.017370 21666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8ff365457ac4791a1e40beaea259c1a: Generated new TSK 0
I20260620 02:57:33.018287 21666 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:57:33.033342 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 21584
W20260620 02:57:33.034857 21646 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.166:44107
W20260620 02:57:33.035720 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 21584: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:57:33.035884 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 21584
W20260620 02:57:33.053318 21569 connection.cc:535] client connection to 127.0.0.1:41073 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41073): Connection reset by peer (error 104)
2026-06-20T02:57:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1993 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:57:33 dist-test-slave-8p8c krb5kdc[21682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:57:33 dist-test-slave-8p8c krb5kdc[21682](info): set up 2 sockets
Jun 20 02:57:33 dist-test-slave-8p8c krb5kdc[21682](info): commencing operation
krb5kdc: starting...
W20260620 02:57:34.166860 388 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.007s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:57:34 dist-test-slave-8p8c krb5kdc[21682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:57:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:57:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:57:34.428247 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37909
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:57:34.760347 21695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:34.760656 21695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:34.760828 21695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:34.768126 21695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:57:34.768249 21695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:34.768335 21695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:34.768451 21695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:57:34.768515 21695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:57:34.778592 21695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:34.779884 21695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:34.781214 21695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:34.793126 21700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:34.793170 21701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:34.793298 21703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:34.793802 21695 server_base.cc:1034] running on GCE node
I20260620 02:57:34.794682 21695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:34.796154 21695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:34.797400 21695 hybrid_clock.cc:648] HybridClock initialized: now 1781924254797351 us; error 53 us; skew 500 ppm
I20260620 02:57:34.797763 21695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:57:34 dist-test-slave-8p8c krb5kdc[21682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:57:34.801020 21695 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:57:34.802490 21695 webserver.cc:460] Webserver started at http://127.0.0.1:41275/ using document root <none> and password file <none>
I20260620 02:57:34.803143 21695 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:34.803243 21695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:34.803486 21695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:34.806438 21695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "02b5deb872fb49ba9f0d91348c476582"
format_stamp: "Formatted at 2026-06-20 02:57:34 on dist-test-slave-8p8c"
server_key: "eae627d8170850271c8d9bd268c0c4ab"
server_key_iv: "85acd5bf768c494567d5ea67d6b3092e"
server_key_version: "encryptionkey@0"
I20260620 02:57:34.807192 21695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "02b5deb872fb49ba9f0d91348c476582"
format_stamp: "Formatted at 2026-06-20 02:57:34 on dist-test-slave-8p8c"
server_key: "eae627d8170850271c8d9bd268c0c4ab"
server_key_iv: "85acd5bf768c494567d5ea67d6b3092e"
server_key_version: "encryptionkey@0"
I20260620 02:57:34.812263 21695 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 02:57:34.815872 21710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:34.817267 21695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 02:57:34.817451 21695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "02b5deb872fb49ba9f0d91348c476582"
format_stamp: "Formatted at 2026-06-20 02:57:34 on dist-test-slave-8p8c"
server_key: "eae627d8170850271c8d9bd268c0c4ab"
server_key_iv: "85acd5bf768c494567d5ea67d6b3092e"
server_key_version: "encryptionkey@0"
I20260620 02:57:34.817657 21695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:34.823343 21695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:34.827692 21695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:34.827976 21695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:34.846504 21695 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37909
I20260620 02:57:34.846541 21761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37909 every 8 connection(s)
I20260620 02:57:34.848006 21695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:57:34.851787 21762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:34.856380 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 21695
I20260620 02:57:34.856575 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:57:34.857082 388 external_mini_cluster.cc:1428] Setting key c0cc0df23d227a0d36a7b1f842eaee81
I20260620 02:57:34.863051 21762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02b5deb872fb49ba9f0d91348c476582: Bootstrap starting.
I20260620 02:57:34.865382 21765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:34.858925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.166:37275 (local address 127.0.0.1:37909)
0620 02:57:34.859808 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:34.859820 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:34.859864 (+ 44us) server_negotiation.cc:407] Connection header received
0620 02:57:34.861778 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:34.861802 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:34.862159 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:34.862570 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:34.863744 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:34.863751 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:34.863764 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:34.863778 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:34.863847 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:34.863870 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:34.864079 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:34.864186 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 02:57:34.864427 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":391,"threads_started":1}
I20260620 02:57:34.866806 21762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02b5deb872fb49ba9f0d91348c476582: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:34.867957 21762 log.cc:824] T 00000000000000000000000000000000 P 02b5deb872fb49ba9f0d91348c476582: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:34.869457 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:57:34.870967 21762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02b5deb872fb49ba9f0d91348c476582: No bootstrap required, opened a new log
I20260620 02:57:34.876770 21765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:34.874021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.166:46341 (local address 127.0.0.1:37909)
0620 02:57:34.874229 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:34.874236 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:34.874649 (+ 413us) server_negotiation.cc:407] Connection header received
0620 02:57:34.874793 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:34.874799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:34.874864 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:34.874979 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:34.875983 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:34.875989 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:34.875996 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:34.876016 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:34.876072 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:34.876094 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:34.876225 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:34.876390 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 02:57:34.876459 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 02:57:34.879586 21762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02b5deb872fb49ba9f0d91348c476582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b5deb872fb49ba9f0d91348c476582" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37909 } }
I20260620 02:57:34.879979 21762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02b5deb872fb49ba9f0d91348c476582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:34.880107 21762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02b5deb872fb49ba9f0d91348c476582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02b5deb872fb49ba9f0d91348c476582, State: Initialized, Role: FOLLOWER
I20260620 02:57:34.880738 21762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02b5deb872fb49ba9f0d91348c476582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b5deb872fb49ba9f0d91348c476582" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37909 } }
I20260620 02:57:34.880976 21762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02b5deb872fb49ba9f0d91348c476582 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:57:34.881132 21762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02b5deb872fb49ba9f0d91348c476582 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:57:34.881304 21762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02b5deb872fb49ba9f0d91348c476582 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:34.883713 21762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02b5deb872fb49ba9f0d91348c476582 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b5deb872fb49ba9f0d91348c476582" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37909 } }
I20260620 02:57:34.884233 21762 leader_election.cc:302] T 00000000000000000000000000000000 P 02b5deb872fb49ba9f0d91348c476582 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02b5deb872fb49ba9f0d91348c476582; no voters:
I20260620 02:57:34.884835 21762 leader_election.cc:288] T 00000000000000000000000000000000 P 02b5deb872fb49ba9f0d91348c476582 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:57:34.885047 21772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02b5deb872fb49ba9f0d91348c476582 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:57:34.885763 21772 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02b5deb872fb49ba9f0d91348c476582 [term 1 LEADER]: Becoming Leader. State: Replica: 02b5deb872fb49ba9f0d91348c476582, State: Running, Role: LEADER
I20260620 02:57:34.886504 21772 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02b5deb872fb49ba9f0d91348c476582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b5deb872fb49ba9f0d91348c476582" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37909 } }
I20260620 02:57:34.887280 21762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02b5deb872fb49ba9f0d91348c476582 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:57:34.889427 21773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02b5deb872fb49ba9f0d91348c476582 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02b5deb872fb49ba9f0d91348c476582" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b5deb872fb49ba9f0d91348c476582" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37909 } } }
I20260620 02:57:34.890126 21773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02b5deb872fb49ba9f0d91348c476582 [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:34.889810 21774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02b5deb872fb49ba9f0d91348c476582 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02b5deb872fb49ba9f0d91348c476582. Latest consensus state: current_term: 1 leader_uuid: "02b5deb872fb49ba9f0d91348c476582" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b5deb872fb49ba9f0d91348c476582" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37909 } } }
I20260620 02:57:34.890976 21774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02b5deb872fb49ba9f0d91348c476582 [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:34.891254 21778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:57:34.899457 21778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:57:34.913422 21778 catalog_manager.cc:1347] Generated new cluster ID: 2e81c45b82f44f0f98d50733f7668437
I20260620 02:57:34.913568 21778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:57:34.938336 21778 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:57:34.938959 21778 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:57:34.950766 21778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02b5deb872fb49ba9f0d91348c476582: Generated new TSK 0
I20260620 02:57:34.951714 21778 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:57:34.959805 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 21695
W20260620 02:57:34.960909 21757 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.166:37275
W20260620 02:57:34.961683 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 21695: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:57:34.961858 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 21695
2026-06-20T02:57:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1922 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:57:35 dist-test-slave-8p8c krb5kdc[21793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:57:35 dist-test-slave-8p8c krb5kdc[21793](info): set up 2 sockets
Jun 20 02:57:35 dist-test-slave-8p8c krb5kdc[21793](info): commencing operation
krb5kdc: starting...
W20260620 02:57:36.074904 388 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:57:36 dist-test-slave-8p8c krb5kdc[21793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:57:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:57:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:57:36.350886 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43851
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:57:36.682399 21806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:36.682788 21806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:36.682972 21806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:36.690264 21806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:57:36.690454 21806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:36.690568 21806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:36.690686 21806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:57:36.690775 21806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:57:36.700554 21806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43851
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:36.701916 21806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:36.703310 21806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:36.716470 21812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:36.716549 21811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:36.716714 21814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:36.718011 21806 server_base.cc:1034] running on GCE node
I20260620 02:57:36.718884 21806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:36.720377 21806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:36.721632 21806 hybrid_clock.cc:648] HybridClock initialized: now 1781924256721578 us; error 44 us; skew 500 ppm
I20260620 02:57:36.722034 21806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:57:36 dist-test-slave-8p8c krb5kdc[21793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:57:36.731208 21806 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:57:36.732838 21806 webserver.cc:460] Webserver started at http://127.0.0.1:43903/ using document root <none> and password file <none>
I20260620 02:57:36.733525 21806 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:36.733635 21806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:36.733901 21806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:36.737016 21806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "53ecd2583dd6405f91b4deab15fe26d1"
format_stamp: "Formatted at 2026-06-20 02:57:36 on dist-test-slave-8p8c"
server_key: "cd0b2cd76262b50312a9f782457ef7e3"
server_key_iv: "b3ffa84590bb30a05e7a7b0f7a161eea"
server_key_version: "encryptionkey@0"
I20260620 02:57:36.737777 21806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "53ecd2583dd6405f91b4deab15fe26d1"
format_stamp: "Formatted at 2026-06-20 02:57:36 on dist-test-slave-8p8c"
server_key: "cd0b2cd76262b50312a9f782457ef7e3"
server_key_iv: "b3ffa84590bb30a05e7a7b0f7a161eea"
server_key_version: "encryptionkey@0"
I20260620 02:57:36.743059 21806 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 02:57:36.746496 21821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:36.748078 21806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 02:57:36.748245 21806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "53ecd2583dd6405f91b4deab15fe26d1"
format_stamp: "Formatted at 2026-06-20 02:57:36 on dist-test-slave-8p8c"
server_key: "cd0b2cd76262b50312a9f782457ef7e3"
server_key_iv: "b3ffa84590bb30a05e7a7b0f7a161eea"
server_key_version: "encryptionkey@0"
I20260620 02:57:36.748412 21806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:36.768448 21806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:36.773036 21806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:36.773293 21806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:36.792529 21806 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43851
I20260620 02:57:36.792621 21872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43851 every 8 connection(s)
I20260620 02:57:36.793958 21806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:57:36.798273 21873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:36.799314 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 21806
I20260620 02:57:36.799531 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:57:36.799954 388 external_mini_cluster.cc:1428] Setting key e72106fd48489f293883dda86f54ddc9
I20260620 02:57:36.812939 21873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53ecd2583dd6405f91b4deab15fe26d1: Bootstrap starting.
Jun 20 02:57:36 dist-test-slave-8p8c krb5kdc[21793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 02:57:36.816654 21873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53ecd2583dd6405f91b4deab15fe26d1: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:36.817900 21873 log.cc:824] T 00000000000000000000000000000000 P 53ecd2583dd6405f91b4deab15fe26d1: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:36.821599 21873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53ecd2583dd6405f91b4deab15fe26d1: No bootstrap required, opened a new log
I20260620 02:57:36.825083 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:36.801919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.166:46025 (local address 127.0.0.1:43851)
0620 02:57:36.802748 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:36.802760 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:36.802799 (+ 39us) server_negotiation.cc:407] Connection header received
0620 02:57:36.804459 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:36.804480 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:36.804895 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:36.805235 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:36.807083 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:36.808406 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:36.810505 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:36.811321 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:36.811481 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:36.816565 (+ 5084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:36.816575 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:36.816592 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:36.816629 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:36.820346 (+ 3717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:36.821392 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:36.821398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:36.821405 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:36.821579 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:36.822083 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:36.822095 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:36.822097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:36.822729 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:36.823082 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:36.823662 (+ 580us) server_negotiation.cc:299] Negotiation successful
0620 02:57:36.823997 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":604,"thread_start_us":378,"threads_started":1}
I20260620 02:57:36.829326 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:57:36.830298 21873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53ecd2583dd6405f91b4deab15fe26d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ecd2583dd6405f91b4deab15fe26d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43851 } }
I20260620 02:57:36.830777 21873 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53ecd2583dd6405f91b4deab15fe26d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:36.830889 21873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53ecd2583dd6405f91b4deab15fe26d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53ecd2583dd6405f91b4deab15fe26d1, State: Initialized, Role: FOLLOWER
I20260620 02:57:36.831501 21873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53ecd2583dd6405f91b4deab15fe26d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ecd2583dd6405f91b4deab15fe26d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43851 } }
I20260620 02:57:36.831853 21873 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53ecd2583dd6405f91b4deab15fe26d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:57:36.832005 21873 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53ecd2583dd6405f91b4deab15fe26d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:57:36.832203 21873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53ecd2583dd6405f91b4deab15fe26d1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:36.835245 21873 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53ecd2583dd6405f91b4deab15fe26d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ecd2583dd6405f91b4deab15fe26d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43851 } }
I20260620 02:57:36.835822 21873 leader_election.cc:302] T 00000000000000000000000000000000 P 53ecd2583dd6405f91b4deab15fe26d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53ecd2583dd6405f91b4deab15fe26d1; no voters:
I20260620 02:57:36.836485 21873 leader_election.cc:288] T 00000000000000000000000000000000 P 53ecd2583dd6405f91b4deab15fe26d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:57:36.836581 21883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53ecd2583dd6405f91b4deab15fe26d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:57:36.837929 21883 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53ecd2583dd6405f91b4deab15fe26d1 [term 1 LEADER]: Becoming Leader. State: Replica: 53ecd2583dd6405f91b4deab15fe26d1, State: Running, Role: LEADER
I20260620 02:57:36.838616 21883 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53ecd2583dd6405f91b4deab15fe26d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ecd2583dd6405f91b4deab15fe26d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43851 } }
I20260620 02:57:36.839051 21873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53ecd2583dd6405f91b4deab15fe26d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:57:36.842800 21884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53ecd2583dd6405f91b4deab15fe26d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53ecd2583dd6405f91b4deab15fe26d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ecd2583dd6405f91b4deab15fe26d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43851 } } }
I20260620 02:57:36.842859 21885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53ecd2583dd6405f91b4deab15fe26d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53ecd2583dd6405f91b4deab15fe26d1. Latest consensus state: current_term: 1 leader_uuid: "53ecd2583dd6405f91b4deab15fe26d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ecd2583dd6405f91b4deab15fe26d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43851 } } }
I20260620 02:57:36.843227 21884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53ecd2583dd6405f91b4deab15fe26d1 [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:36.843255 21885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53ecd2583dd6405f91b4deab15fe26d1 [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:36.854774 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:36.834199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.166:54053 (local address 127.0.0.1:43851)
0620 02:57:36.834420 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:36.834427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:36.834741 (+ 314us) server_negotiation.cc:407] Connection header received
0620 02:57:36.834879 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:36.834884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:36.834971 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:36.835088 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:36.836891 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:36.838087 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:36.839641 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:36.840159 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:36.840303 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:36.843987 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:36.843998 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:36.844006 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:36.844058 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:36.848823 (+ 4765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:36.850237 (+ 1414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:36.850245 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:36.850252 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:36.850459 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:36.851202 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:36.851210 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:36.851216 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:36.851921 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:36.853770 (+ 1849us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:36.853997 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 02:57:36.854214 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 02:57:36.858927 21891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:57:36.864670 21891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:57:36.874524 21891 catalog_manager.cc:1347] Generated new cluster ID: 9bb403a0445743838075526353d6b385
I20260620 02:57:36.874697 21891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:57:36.902206 21891 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:57:36.903697 21891 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:57:36.912701 21891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53ecd2583dd6405f91b4deab15fe26d1: Generated new TSK 0
I20260620 02:57:36.913980 21891 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:57:36.931061 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 21806
I20260620 02:57:36.932612 21868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:57:37.028370 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 21806
W20260620 02:57:37.043601 21791 connection.cc:535] client connection to 127.0.0.1:43851 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43851): Connection reset by peer (error 104)
2026-06-20T02:57:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2067 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:57:37 dist-test-slave-8p8c krb5kdc[21905](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:57:37 dist-test-slave-8p8c krb5kdc[21905](info): set up 2 sockets
Jun 20 02:57:37 dist-test-slave-8p8c krb5kdc[21905](info): commencing operation
krb5kdc: starting...
W20260620 02:57:38.142784 388 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.007s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:57:38 dist-test-slave-8p8c krb5kdc[21905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:57:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:57:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:57:38.405870 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33527
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:57:38.752173 21918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:38.752486 21918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:38.752581 21918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:38.759905 21918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:57:38.760032 21918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:38.760079 21918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:38.760138 21918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:57:38.760183 21918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:57:38.770922 21918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33527
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:38.772354 21918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:38.773660 21918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:38.786424 21923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:38.786327 21924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:38.786583 21926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:38.787228 21918 server_base.cc:1034] running on GCE node
I20260620 02:57:38.788357 21918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:38.790057 21918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:38.791330 21918 hybrid_clock.cc:648] HybridClock initialized: now 1781924258791252 us; error 67 us; skew 500 ppm
I20260620 02:57:38.791747 21918 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:57:38 dist-test-slave-8p8c krb5kdc[21905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:57:38.795107 21918 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:57:38.797071 21918 webserver.cc:460] Webserver started at http://127.0.0.1:33819/ using document root <none> and password file <none>
I20260620 02:57:38.797871 21918 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:38.798020 21918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:38.798341 21918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:38.801447 21918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "2b22934befb94d27a7118d24d536c14d"
format_stamp: "Formatted at 2026-06-20 02:57:38 on dist-test-slave-8p8c"
server_key: "12e526040ea02f193886c6e381b6283f"
server_key_iv: "8350f5e694f7c543fe29bb4df46fdd13"
server_key_version: "encryptionkey@0"
I20260620 02:57:38.802256 21918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "2b22934befb94d27a7118d24d536c14d"
format_stamp: "Formatted at 2026-06-20 02:57:38 on dist-test-slave-8p8c"
server_key: "12e526040ea02f193886c6e381b6283f"
server_key_iv: "8350f5e694f7c543fe29bb4df46fdd13"
server_key_version: "encryptionkey@0"
I20260620 02:57:38.807638 21918 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 02:57:38.811326 21933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:38.812927 21918 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 02:57:38.813130 21918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "2b22934befb94d27a7118d24d536c14d"
format_stamp: "Formatted at 2026-06-20 02:57:38 on dist-test-slave-8p8c"
server_key: "12e526040ea02f193886c6e381b6283f"
server_key_iv: "8350f5e694f7c543fe29bb4df46fdd13"
server_key_version: "encryptionkey@0"
I20260620 02:57:38.813344 21918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:38.832504 21918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:38.836817 21918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:38.837047 21918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:38.856505 21918 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33527
I20260620 02:57:38.856596 21984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33527 every 8 connection(s)
I20260620 02:57:38.857885 21918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:57:38.861603 21985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:38.863404 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 21918
I20260620 02:57:38.863641 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:57:38.864097 388 external_mini_cluster.cc:1428] Setting key 38cf0c2e248a053312acecc9ab9c0215
I20260620 02:57:38.874725 21985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b22934befb94d27a7118d24d536c14d: Bootstrap starting.
I20260620 02:57:38.877485 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:38.865855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.166:34915 (local address 127.0.0.1:33527)
0620 02:57:38.866787 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:38.866800 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:38.866846 (+ 46us) server_negotiation.cc:407] Connection header received
0620 02:57:38.868672 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:38.868696 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:38.869122 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:38.869540 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:38.870065 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:38.871392 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:38.872774 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:38.873443 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:38.873630 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:38.875001 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:38.875009 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:38.875022 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:38.875594 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:38.875674 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:38.875698 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:38.875871 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:38.876284 (+ 413us) server_negotiation.cc:299] Negotiation successful
0620 02:57:38.876470 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":430,"threads_started":1}
I20260620 02:57:38.878316 21985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b22934befb94d27a7118d24d536c14d: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:38.879570 21985 log.cc:824] T 00000000000000000000000000000000 P 2b22934befb94d27a7118d24d536c14d: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:38.881971 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:57:38.882969 21985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b22934befb94d27a7118d24d536c14d: No bootstrap required, opened a new log
I20260620 02:57:38.893816 21985 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b22934befb94d27a7118d24d536c14d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b22934befb94d27a7118d24d536c14d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33527 } }
I20260620 02:57:38.894429 21985 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b22934befb94d27a7118d24d536c14d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:38.894559 21985 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b22934befb94d27a7118d24d536c14d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b22934befb94d27a7118d24d536c14d, State: Initialized, Role: FOLLOWER
I20260620 02:57:38.894743 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:38.887572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.166:49139 (local address 127.0.0.1:33527)
0620 02:57:38.887847 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:38.887857 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:38.888206 (+ 349us) server_negotiation.cc:407] Connection header received
0620 02:57:38.888430 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:38.888434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:38.888505 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:38.888607 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:38.889234 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:38.890021 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:38.891990 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:38.892456 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:38.892560 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:38.893784 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:38.893789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:38.893792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:38.893812 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:38.893850 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:38.893864 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:38.893947 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:38.894190 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 02:57:38.894274 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 02:57:38.895373 21985 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b22934befb94d27a7118d24d536c14d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b22934befb94d27a7118d24d536c14d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33527 } }
I20260620 02:57:38.895639 21985 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b22934befb94d27a7118d24d536c14d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:57:38.895800 21985 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b22934befb94d27a7118d24d536c14d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:57:38.895982 21985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b22934befb94d27a7118d24d536c14d [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:38.898188 21985 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b22934befb94d27a7118d24d536c14d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b22934befb94d27a7118d24d536c14d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33527 } }
I20260620 02:57:38.898748 21985 leader_election.cc:302] T 00000000000000000000000000000000 P 2b22934befb94d27a7118d24d536c14d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b22934befb94d27a7118d24d536c14d; no voters:
I20260620 02:57:38.899377 21985 leader_election.cc:288] T 00000000000000000000000000000000 P 2b22934befb94d27a7118d24d536c14d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:57:38.899564 21995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b22934befb94d27a7118d24d536c14d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:57:38.900367 21995 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b22934befb94d27a7118d24d536c14d [term 1 LEADER]: Becoming Leader. State: Replica: 2b22934befb94d27a7118d24d536c14d, State: Running, Role: LEADER
I20260620 02:57:38.900957 21995 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b22934befb94d27a7118d24d536c14d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b22934befb94d27a7118d24d536c14d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33527 } }
I20260620 02:57:38.901670 21985 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b22934befb94d27a7118d24d536c14d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:57:38.903584 21996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b22934befb94d27a7118d24d536c14d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b22934befb94d27a7118d24d536c14d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b22934befb94d27a7118d24d536c14d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33527 } } }
I20260620 02:57:38.903900 21996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b22934befb94d27a7118d24d536c14d [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:38.904778 22000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:57:38.904709 21997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b22934befb94d27a7118d24d536c14d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b22934befb94d27a7118d24d536c14d. Latest consensus state: current_term: 1 leader_uuid: "2b22934befb94d27a7118d24d536c14d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b22934befb94d27a7118d24d536c14d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33527 } } }
I20260620 02:57:38.905117 21997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b22934befb94d27a7118d24d536c14d [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:38.910351 22000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:57:38.923576 22000 catalog_manager.cc:1347] Generated new cluster ID: a610da8821c44736ab5f5c9afe83043d
I20260620 02:57:38.923712 22000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:57:38.939474 22000 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:57:38.940899 22000 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:57:38.948669 22000 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b22934befb94d27a7118d24d536c14d: Generated new TSK 0
I20260620 02:57:38.949536 22000 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:57:38.957159 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 21918
W20260620 02:57:38.958415 21980 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.166:34915
W20260620 02:57:38.959483 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 21918: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:57:38.959671 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 21918
W20260620 02:57:38.977697 21903 connection.cc:535] client connection to 127.0.0.1:33527 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33527): Connection reset by peer (error 104)
2026-06-20T02:57:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1934 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:57:39 dist-test-slave-8p8c krb5kdc[22016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:57:39 dist-test-slave-8p8c krb5kdc[22016](info): set up 2 sockets
Jun 20 02:57:39 dist-test-slave-8p8c krb5kdc[22016](info): commencing operation
krb5kdc: starting...
W20260620 02:57:40.082865 388 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.011s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:57:40 dist-test-slave-8p8c krb5kdc[22016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:57:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:57:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 02:57:40.338131 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34651
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:57:40.682811 22029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:40.683117 22029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:40.683281 22029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:40.690521 22029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:57:40.690637 22029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:40.690732 22029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:40.690840 22029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:57:40.690917 22029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:57:40.700901 22029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34651
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781923914418765-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:40.702328 22029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:40.703708 22029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:40.715680 22035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:40.715737 22034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:40.715762 22037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:40.716758 22029 server_base.cc:1034] running on GCE node
I20260620 02:57:40.717562 22029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:40.719117 22029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:40.720353 22029 hybrid_clock.cc:648] HybridClock initialized: now 1781924260720313 us; error 27 us; skew 500 ppm
I20260620 02:57:40.720769 22029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:57:40 dist-test-slave-8p8c krb5kdc[22016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:57:40.724167 22029 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 02:57:40.725822 22029 webserver.cc:460] Webserver started at http://127.0.0.1:43721/ using document root <none> and password file <none>
I20260620 02:57:40.726619 22029 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:40.726771 22029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:40.727113 22029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:40.730185 22029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "148ce10aab0347adaac6b6764b207376"
format_stamp: "Formatted at 2026-06-20 02:57:40 on dist-test-slave-8p8c"
server_key: "c43a76999ef86ab9de11aa54ce2b31db"
server_key_iv: "906786c0a0db3e2da45fa1d91672d8a7"
server_key_version: "encryptionkey@0"
I20260620 02:57:40.731046 22029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "148ce10aab0347adaac6b6764b207376"
format_stamp: "Formatted at 2026-06-20 02:57:40 on dist-test-slave-8p8c"
server_key: "c43a76999ef86ab9de11aa54ce2b31db"
server_key_iv: "906786c0a0db3e2da45fa1d91672d8a7"
server_key_version: "encryptionkey@0"
I20260620 02:57:40.736351 22029 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 02:57:40.739971 22044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:40.741345 22029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:57:40.741555 22029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "148ce10aab0347adaac6b6764b207376"
format_stamp: "Formatted at 2026-06-20 02:57:40 on dist-test-slave-8p8c"
server_key: "c43a76999ef86ab9de11aa54ce2b31db"
server_key_iv: "906786c0a0db3e2da45fa1d91672d8a7"
server_key_version: "encryptionkey@0"
I20260620 02:57:40.741799 22029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:40.747676 22029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:40.751875 22029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:40.752183 22029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:40.771186 22029 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34651
I20260620 02:57:40.771207 22095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34651 every 8 connection(s)
I20260620 02:57:40.772668 22029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:57:40.773239 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 22029
I20260620 02:57:40.773447 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:57:40.774111 388 external_mini_cluster.cc:1428] Setting key ee105cb3b4d24093f43b807ee4011bf1
I20260620 02:57:40.777155 22096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:40.782094 22099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:40.776027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.166:59597 (local address 127.0.0.1:34651)
0620 02:57:40.776831 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:40.776842 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:40.776883 (+ 41us) server_negotiation.cc:407] Connection header received
0620 02:57:40.778669 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:40.778692 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:40.779035 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:40.779435 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:40.780491 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:40.780497 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:40.780509 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:40.780523 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:40.780594 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:40.780613 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:40.780872 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:40.781019 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 02:57:40.781194 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":357,"threads_started":1}
I20260620 02:57:40.785928 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:57:40.788951 22096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 148ce10aab0347adaac6b6764b207376: Bootstrap starting.
I20260620 02:57:40.792695 22096 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 148ce10aab0347adaac6b6764b207376: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:40.792804 22099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:40.790333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.166:53127 (local address 127.0.0.1:34651)
0620 02:57:40.790572 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:40.790578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:40.790787 (+ 209us) server_negotiation.cc:407] Connection header received
0620 02:57:40.790907 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:40.790918 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:40.790987 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:40.791093 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:40.792039 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:40.792045 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:40.792049 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:40.792067 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:40.792114 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:40.792135 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:40.792253 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:40.792402 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 02:57:40.792466 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260620 02:57:40.793959 22096 log.cc:824] T 00000000000000000000000000000000 P 148ce10aab0347adaac6b6764b207376: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:40.796980 22096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 148ce10aab0347adaac6b6764b207376: No bootstrap required, opened a new log
I20260620 02:57:40.804423 22096 raft_consensus.cc:348] T 00000000000000000000000000000000 P 148ce10aab0347adaac6b6764b207376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "148ce10aab0347adaac6b6764b207376" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34651 } }
I20260620 02:57:40.804818 22096 raft_consensus.cc:374] T 00000000000000000000000000000000 P 148ce10aab0347adaac6b6764b207376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:40.804941 22096 raft_consensus.cc:729] T 00000000000000000000000000000000 P 148ce10aab0347adaac6b6764b207376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148ce10aab0347adaac6b6764b207376, State: Initialized, Role: FOLLOWER
I20260620 02:57:40.805552 22096 consensus_queue.cc:260] T 00000000000000000000000000000000 P 148ce10aab0347adaac6b6764b207376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "148ce10aab0347adaac6b6764b207376" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34651 } }
I20260620 02:57:40.805754 22096 raft_consensus.cc:388] T 00000000000000000000000000000000 P 148ce10aab0347adaac6b6764b207376 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:57:40.805891 22096 raft_consensus.cc:482] T 00000000000000000000000000000000 P 148ce10aab0347adaac6b6764b207376 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:57:40.806057 22096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 148ce10aab0347adaac6b6764b207376 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:40.808544 22096 raft_consensus.cc:504] T 00000000000000000000000000000000 P 148ce10aab0347adaac6b6764b207376 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "148ce10aab0347adaac6b6764b207376" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34651 } }
I20260620 02:57:40.809041 22096 leader_election.cc:302] T 00000000000000000000000000000000 P 148ce10aab0347adaac6b6764b207376 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 148ce10aab0347adaac6b6764b207376; no voters:
I20260620 02:57:40.809623 22096 leader_election.cc:288] T 00000000000000000000000000000000 P 148ce10aab0347adaac6b6764b207376 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:57:40.809772 22106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 148ce10aab0347adaac6b6764b207376 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:57:40.810504 22106 raft_consensus.cc:686] T 00000000000000000000000000000000 P 148ce10aab0347adaac6b6764b207376 [term 1 LEADER]: Becoming Leader. State: Replica: 148ce10aab0347adaac6b6764b207376, State: Running, Role: LEADER
I20260620 02:57:40.811043 22106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 148ce10aab0347adaac6b6764b207376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "148ce10aab0347adaac6b6764b207376" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34651 } }
I20260620 02:57:40.811707 22096 sys_catalog.cc:564] T 00000000000000000000000000000000 P 148ce10aab0347adaac6b6764b207376 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:57:40.813859 22108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 148ce10aab0347adaac6b6764b207376 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 148ce10aab0347adaac6b6764b207376. Latest consensus state: current_term: 1 leader_uuid: "148ce10aab0347adaac6b6764b207376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "148ce10aab0347adaac6b6764b207376" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34651 } } }
I20260620 02:57:40.814193 22108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 148ce10aab0347adaac6b6764b207376 [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:40.814355 22107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 148ce10aab0347adaac6b6764b207376 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "148ce10aab0347adaac6b6764b207376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "148ce10aab0347adaac6b6764b207376" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34651 } } }
I20260620 02:57:40.814667 22107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 148ce10aab0347adaac6b6764b207376 [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:40.816425 22111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:57:40.821771 22111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:57:40.830950 22111 catalog_manager.cc:1347] Generated new cluster ID: 89df83ae89ff491285f9377c77b827c7
I20260620 02:57:40.831059 22111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:57:40.854058 22111 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:57:40.854583 22111 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:57:40.865195 22111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 148ce10aab0347adaac6b6764b207376: Generated new TSK 0
I20260620 02:57:40.865999 22111 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:57:40.872582 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 22029
W20260620 02:57:40.873710 22091 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.166:59597
W20260620 02:57:40.874450 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 22029: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:57:40.874604 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 22029
2026-06-20T02:57:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1912 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:57:40 dist-test-slave-8p8c krb5kdc[22127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:57:40 dist-test-slave-8p8c krb5kdc[22127](info): set up 2 sockets
Jun 20 02:57:40 dist-test-slave-8p8c krb5kdc[22127](info): commencing operation
krb5kdc: starting...
W20260620 02:57:42.031505 388 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.009s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:57:42 dist-test-slave-8p8c krb5kdc[22127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:57:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:57:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:57:42.297852 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41331
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:57:42.635844 22140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:42.636143 22140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:42.636242 22140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:42.643478 22140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:57:42.643594 22140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:42.643641 22140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:42.643708 22140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:57:42.643754 22140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:57:42.653611 22140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41331
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:42.654872 22140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:42.656136 22140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:42.668080 22146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:42.668087 22148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:42.668346 22145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:42.669369 22140 server_base.cc:1034] running on GCE node
I20260620 02:57:42.670150 22140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:42.671772 22140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:42.673003 22140 hybrid_clock.cc:648] HybridClock initialized: now 1781924262672953 us; error 37 us; skew 500 ppm
I20260620 02:57:42.673400 22140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:57:42 dist-test-slave-8p8c krb5kdc[22127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924262, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:57:42.686219 22140 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:57:42.687965 22140 webserver.cc:460] Webserver started at http://127.0.97.62:45063/ using document root <none> and password file <none>
I20260620 02:57:42.688725 22140 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:42.688877 22140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:42.689204 22140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:42.692412 22140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "b8477a8e09774c05a6462009be6f3a5e"
format_stamp: "Formatted at 2026-06-20 02:57:42 on dist-test-slave-8p8c"
server_key: "818db928eef51363b47848db50d9f2da"
server_key_iv: "fc82670ababcfd8d3a7a7a04ee66426d"
server_key_version: "encryptionkey@0"
I20260620 02:57:42.693195 22140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "b8477a8e09774c05a6462009be6f3a5e"
format_stamp: "Formatted at 2026-06-20 02:57:42 on dist-test-slave-8p8c"
server_key: "818db928eef51363b47848db50d9f2da"
server_key_iv: "fc82670ababcfd8d3a7a7a04ee66426d"
server_key_version: "encryptionkey@0"
I20260620 02:57:42.698585 22140 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 02:57:42.701928 22155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:42.703404 22140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:57:42.703608 22140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "b8477a8e09774c05a6462009be6f3a5e"
format_stamp: "Formatted at 2026-06-20 02:57:42 on dist-test-slave-8p8c"
server_key: "818db928eef51363b47848db50d9f2da"
server_key_iv: "fc82670ababcfd8d3a7a7a04ee66426d"
server_key_version: "encryptionkey@0"
I20260620 02:57:42.703817 22140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:42.735822 22140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:42.740723 22140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:42.741030 22140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:42.761127 22140 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41331
I20260620 02:57:42.761220 22206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41331 every 8 connection(s)
I20260620 02:57:42.762600 22140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:57:42.763209 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 22140
I20260620 02:57:42.763434 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:57:42.764065 388 external_mini_cluster.cc:1428] Setting key aba79302c4df39499e5262f17af3d8f0
I20260620 02:57:42.767562 22207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 02:57:42 dist-test-slave-8p8c krb5kdc[22127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:57:42.779345 22207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8477a8e09774c05a6462009be6f3a5e: Bootstrap starting.
I20260620 02:57:42.782910 22207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8477a8e09774c05a6462009be6f3a5e: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:42.784003 22207 log.cc:824] T 00000000000000000000000000000000 P b8477a8e09774c05a6462009be6f3a5e: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:42.787415 22207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8477a8e09774c05a6462009be6f3a5e: No bootstrap required, opened a new log
I20260620 02:57:42.788802 22210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:42.765942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54645 (local address 127.0.97.62:41331)
0620 02:57:42.766962 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:42.766976 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:42.767024 (+ 48us) server_negotiation.cc:407] Connection header received
0620 02:57:42.768824 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:42.768847 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:42.769249 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:42.769672 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:42.771353 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:42.772802 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:42.774181 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:42.774971 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:42.775177 (+ 206us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:42.780387 (+ 5210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:42.780396 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:42.780411 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:42.780458 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:42.783818 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:42.784848 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:42.784854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:42.784860 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:42.785083 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:42.785593 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:42.785597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:42.785600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:42.786259 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:42.786600 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:42.787113 (+ 513us) server_negotiation.cc:299] Negotiation successful
0620 02:57:42.787452 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":439,"threads_started":1}
I20260620 02:57:42.792847 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:57:42.797226 22207 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8477a8e09774c05a6462009be6f3a5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8477a8e09774c05a6462009be6f3a5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41331 } }
I20260620 02:57:42.797685 22207 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8477a8e09774c05a6462009be6f3a5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:42.797827 22207 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8477a8e09774c05a6462009be6f3a5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8477a8e09774c05a6462009be6f3a5e, State: Initialized, Role: FOLLOWER
I20260620 02:57:42.798583 22207 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8477a8e09774c05a6462009be6f3a5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8477a8e09774c05a6462009be6f3a5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41331 } }
I20260620 02:57:42.798880 22207 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8477a8e09774c05a6462009be6f3a5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:57:42.799034 22207 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8477a8e09774c05a6462009be6f3a5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:57:42.799242 22207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8477a8e09774c05a6462009be6f3a5e [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:42.801416 22207 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8477a8e09774c05a6462009be6f3a5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8477a8e09774c05a6462009be6f3a5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41331 } }
I20260620 02:57:42.801935 22207 leader_election.cc:302] T 00000000000000000000000000000000 P b8477a8e09774c05a6462009be6f3a5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8477a8e09774c05a6462009be6f3a5e; no voters:
I20260620 02:57:42.802634 22207 leader_election.cc:288] T 00000000000000000000000000000000 P b8477a8e09774c05a6462009be6f3a5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:57:42.802736 22217 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8477a8e09774c05a6462009be6f3a5e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:57:42.805269 22207 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8477a8e09774c05a6462009be6f3a5e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:57:42.806700 22217 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8477a8e09774c05a6462009be6f3a5e [term 1 LEADER]: Becoming Leader. State: Replica: b8477a8e09774c05a6462009be6f3a5e, State: Running, Role: LEADER
I20260620 02:57:42.807586 22217 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8477a8e09774c05a6462009be6f3a5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8477a8e09774c05a6462009be6f3a5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41331 } }
I20260620 02:57:42.811353 22218 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8477a8e09774c05a6462009be6f3a5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8477a8e09774c05a6462009be6f3a5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8477a8e09774c05a6462009be6f3a5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41331 } } }
I20260620 02:57:42.811789 22218 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8477a8e09774c05a6462009be6f3a5e [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:42.811329 22219 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8477a8e09774c05a6462009be6f3a5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8477a8e09774c05a6462009be6f3a5e. Latest consensus state: current_term: 1 leader_uuid: "b8477a8e09774c05a6462009be6f3a5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8477a8e09774c05a6462009be6f3a5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41331 } } }
I20260620 02:57:42.813390 22219 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8477a8e09774c05a6462009be6f3a5e [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:42.815303 22210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:42.798260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60155 (local address 127.0.97.62:41331)
0620 02:57:42.798518 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:42.798528 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:42.798991 (+ 463us) server_negotiation.cc:407] Connection header received
0620 02:57:42.799199 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:42.799205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:42.799281 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:42.799396 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:42.800925 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:42.801770 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:42.803402 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:42.804014 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:42.804092 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:42.806972 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:42.806977 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:42.806980 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:42.807008 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:42.810420 (+ 3412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:42.811466 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:42.811471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:42.811474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:42.811730 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:42.812401 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:42.812409 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:42.812415 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:42.812913 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:42.813250 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:42.814564 (+ 1314us) server_negotiation.cc:299] Negotiation successful
0620 02:57:42.814802 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 02:57:42.825479 22223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:57:42.830579 22223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:57:42.839258 22223 catalog_manager.cc:1347] Generated new cluster ID: 8d3afea50ae042f48168b126c8519d2e
I20260620 02:57:42.839555 22223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:57:42.864146 22223 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:57:42.865540 22223 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:57:42.875317 22223 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8477a8e09774c05a6462009be6f3a5e: Generated new TSK 0
I20260620 02:57:42.876253 22223 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:57:42.888458 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 22140
I20260620 02:57:42.889667 22202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:57:42.988790 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 22140
W20260620 02:57:43.007099 22125 connection.cc:535] client connection to 127.0.97.62:41331 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:41331): Connection reset by peer (error 104)
2026-06-20T02:57:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2116 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:57:43 dist-test-slave-8p8c krb5kdc[22239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:57:43 dist-test-slave-8p8c krb5kdc[22239](info): set up 2 sockets
Jun 20 02:57:43 dist-test-slave-8p8c krb5kdc[22239](info): commencing operation
krb5kdc: starting...
W20260620 02:57:44.138808 388 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.008s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:57:44 dist-test-slave-8p8c krb5kdc[22239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:57:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:57:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:57:44.402096 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37211
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:57:44.735796 22252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:44.736100 22252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:44.736194 22252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:44.743577 22252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:57:44.743714 22252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:44.743765 22252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:44.743903 22252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:57:44.744021 22252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:57:44.755394 22252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37211
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:44.756743 22252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:44.758096 22252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:44.770972 22257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:44.770957 22258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:44.771044 22260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:44.772374 22252 server_base.cc:1034] running on GCE node
I20260620 02:57:44.773252 22252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:44.774833 22252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:44.776126 22252 hybrid_clock.cc:648] HybridClock initialized: now 1781924264776072 us; error 45 us; skew 500 ppm
I20260620 02:57:44.776521 22252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:57:44 dist-test-slave-8p8c krb5kdc[22239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:57:44.779861 22252 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:57:44.781452 22252 webserver.cc:460] Webserver started at http://127.0.97.62:41901/ using document root <none> and password file <none>
I20260620 02:57:44.782145 22252 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:44.782245 22252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:44.782727 22252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:44.785847 22252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "80fd64f468b54ccfaaee9a45a604d89c"
format_stamp: "Formatted at 2026-06-20 02:57:44 on dist-test-slave-8p8c"
server_key: "3edf5c99c282347a417b0c470c5710ea"
server_key_iv: "1a3ba46f156c42a1e5e63acae7703829"
server_key_version: "encryptionkey@0"
I20260620 02:57:44.786705 22252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "80fd64f468b54ccfaaee9a45a604d89c"
format_stamp: "Formatted at 2026-06-20 02:57:44 on dist-test-slave-8p8c"
server_key: "3edf5c99c282347a417b0c470c5710ea"
server_key_iv: "1a3ba46f156c42a1e5e63acae7703829"
server_key_version: "encryptionkey@0"
I20260620 02:57:44.792261 22252 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 02:57:44.796232 22267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:44.797829 22252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:57:44.797991 22252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "80fd64f468b54ccfaaee9a45a604d89c"
format_stamp: "Formatted at 2026-06-20 02:57:44 on dist-test-slave-8p8c"
server_key: "3edf5c99c282347a417b0c470c5710ea"
server_key_iv: "1a3ba46f156c42a1e5e63acae7703829"
server_key_version: "encryptionkey@0"
I20260620 02:57:44.798250 22252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:44.810927 22252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:44.815724 22252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:44.816040 22252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:44.836169 22252 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37211
I20260620 02:57:44.836263 22318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37211 every 8 connection(s)
I20260620 02:57:44.837664 22252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:57:44.841629 22319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:44.846937 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 22252
I20260620 02:57:44.847157 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:57:44.847594 388 external_mini_cluster.cc:1428] Setting key 14f576b3e8a81e506b51266d267d3ac0
I20260620 02:57:44.853003 22319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80fd64f468b54ccfaaee9a45a604d89c: Bootstrap starting.
I20260620 02:57:44.856699 22319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80fd64f468b54ccfaaee9a45a604d89c: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:44.857838 22319 log.cc:824] T 00000000000000000000000000000000 P 80fd64f468b54ccfaaee9a45a604d89c: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:44.861115 22319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80fd64f468b54ccfaaee9a45a604d89c: No bootstrap required, opened a new log
I20260620 02:57:44.862017 22322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:44.849374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43585 (local address 127.0.97.62:37211)
0620 02:57:44.850244 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:44.850259 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:44.850305 (+ 46us) server_negotiation.cc:407] Connection header received
0620 02:57:44.852164 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:44.852189 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:44.852606 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:44.852923 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:44.853560 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:44.855032 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:44.856933 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:44.857645 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:44.857786 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:44.859315 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:44.859328 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:44.859354 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:44.859876 (+ 522us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:44.859951 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:44.859980 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:44.860098 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:44.860472 (+ 374us) server_negotiation.cc:299] Negotiation successful
0620 02:57:44.860671 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":414,"threads_started":1}
I20260620 02:57:44.866526 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:57:44.872417 22319 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80fd64f468b54ccfaaee9a45a604d89c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80fd64f468b54ccfaaee9a45a604d89c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37211 } }
I20260620 02:57:44.873005 22319 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80fd64f468b54ccfaaee9a45a604d89c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:44.873145 22319 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80fd64f468b54ccfaaee9a45a604d89c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd64f468b54ccfaaee9a45a604d89c, State: Initialized, Role: FOLLOWER
I20260620 02:57:44.874176 22319 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80fd64f468b54ccfaaee9a45a604d89c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80fd64f468b54ccfaaee9a45a604d89c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37211 } }
I20260620 02:57:44.874583 22319 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80fd64f468b54ccfaaee9a45a604d89c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:57:44.874774 22319 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80fd64f468b54ccfaaee9a45a604d89c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:57:44.874989 22319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80fd64f468b54ccfaaee9a45a604d89c [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:44.878577 22319 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80fd64f468b54ccfaaee9a45a604d89c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80fd64f468b54ccfaaee9a45a604d89c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37211 } }
I20260620 02:57:44.879338 22319 leader_election.cc:302] T 00000000000000000000000000000000 P 80fd64f468b54ccfaaee9a45a604d89c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80fd64f468b54ccfaaee9a45a604d89c; no voters:
I20260620 02:57:44.880084 22322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:44.872105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55391 (local address 127.0.97.62:37211)
0620 02:57:44.872271 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:44.872276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:44.872682 (+ 406us) server_negotiation.cc:407] Connection header received
0620 02:57:44.872946 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:44.872951 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:44.873040 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:44.873173 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:44.873964 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:44.875089 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:44.877201 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:44.877655 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:44.877756 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:44.879119 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:44.879126 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:44.879131 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:44.879152 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:44.879198 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:44.879215 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:44.879301 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:44.879644 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 02:57:44.879707 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 02:57:44.880198 22319 leader_election.cc:288] T 00000000000000000000000000000000 P 80fd64f468b54ccfaaee9a45a604d89c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:57:44.880403 22329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80fd64f468b54ccfaaee9a45a604d89c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:57:44.882021 22329 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80fd64f468b54ccfaaee9a45a604d89c [term 1 LEADER]: Becoming Leader. State: Replica: 80fd64f468b54ccfaaee9a45a604d89c, State: Running, Role: LEADER
I20260620 02:57:44.882748 22329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80fd64f468b54ccfaaee9a45a604d89c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80fd64f468b54ccfaaee9a45a604d89c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37211 } }
I20260620 02:57:44.883018 22319 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80fd64f468b54ccfaaee9a45a604d89c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:57:44.886055 22331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80fd64f468b54ccfaaee9a45a604d89c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80fd64f468b54ccfaaee9a45a604d89c. Latest consensus state: current_term: 1 leader_uuid: "80fd64f468b54ccfaaee9a45a604d89c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80fd64f468b54ccfaaee9a45a604d89c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37211 } } }
I20260620 02:57:44.886492 22331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80fd64f468b54ccfaaee9a45a604d89c [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:44.886253 22330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80fd64f468b54ccfaaee9a45a604d89c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80fd64f468b54ccfaaee9a45a604d89c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80fd64f468b54ccfaaee9a45a604d89c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37211 } } }
I20260620 02:57:44.886715 22330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80fd64f468b54ccfaaee9a45a604d89c [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:44.892081 22335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:57:44.898263 22335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:57:44.907847 22335 catalog_manager.cc:1347] Generated new cluster ID: 3e798a9d27e145edbaf1caa4d0ca30c3
I20260620 02:57:44.908022 22335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:57:44.917739 22335 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:57:44.919884 22335 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:57:44.930538 22335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80fd64f468b54ccfaaee9a45a604d89c: Generated new TSK 0
I20260620 02:57:44.931737 22335 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:57:44.946571 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 22252
W20260620 02:57:44.948138 22314 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43585
W20260620 02:57:44.949040 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 22252: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:57:44.949263 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 22252
W20260620 02:57:44.964502 22237 connection.cc:535] client connection to 127.0.97.62:37211 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:37211): Connection reset by peer (error 104)
2026-06-20T02:57:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1957 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:57:45 dist-test-slave-8p8c krb5kdc[22351](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:57:45 dist-test-slave-8p8c krb5kdc[22351](info): set up 2 sockets
Jun 20 02:57:45 dist-test-slave-8p8c krb5kdc[22351](info): commencing operation
krb5kdc: starting...
W20260620 02:57:46.066962 388 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:57:46 dist-test-slave-8p8c krb5kdc[22351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:57:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:57:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:57:46.327878 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38213
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:57:46.670970 22364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:46.671283 22364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:46.671396 22364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:46.678735 22364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:57:46.678875 22364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:46.678936 22364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:46.679008 22364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:57:46.679062 22364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:57:46.689193 22364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38213
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:46.690584 22364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:46.692046 22364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:46.705318 22370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:46.705273 22369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:46.705862 22372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:46.706172 22364 server_base.cc:1034] running on GCE node
I20260620 02:57:46.707096 22364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:46.708593 22364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:46.709836 22364 hybrid_clock.cc:648] HybridClock initialized: now 1781924266709793 us; error 55 us; skew 500 ppm
I20260620 02:57:46.710199 22364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:57:46 dist-test-slave-8p8c krb5kdc[22351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:57:46.713474 22364 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:57:46.715153 22364 webserver.cc:460] Webserver started at http://127.0.97.62:45423/ using document root <none> and password file <none>
I20260620 02:57:46.715859 22364 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:46.716002 22364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:46.716307 22364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:46.719357 22364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "2e7bb86e7b2542eb9070e0ba1cbe4fdd"
format_stamp: "Formatted at 2026-06-20 02:57:46 on dist-test-slave-8p8c"
server_key: "9912df2a09b15949e64a14685149094a"
server_key_iv: "4939bbc0c1ecb4b2c92edeb005e2e466"
server_key_version: "encryptionkey@0"
I20260620 02:57:46.720129 22364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "2e7bb86e7b2542eb9070e0ba1cbe4fdd"
format_stamp: "Formatted at 2026-06-20 02:57:46 on dist-test-slave-8p8c"
server_key: "9912df2a09b15949e64a14685149094a"
server_key_iv: "4939bbc0c1ecb4b2c92edeb005e2e466"
server_key_version: "encryptionkey@0"
I20260620 02:57:46.725427 22364 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:57:46.729192 22379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:46.730683 22364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:57:46.730867 22364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "2e7bb86e7b2542eb9070e0ba1cbe4fdd"
format_stamp: "Formatted at 2026-06-20 02:57:46 on dist-test-slave-8p8c"
server_key: "9912df2a09b15949e64a14685149094a"
server_key_iv: "4939bbc0c1ecb4b2c92edeb005e2e466"
server_key_version: "encryptionkey@0"
I20260620 02:57:46.731071 22364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:46.736901 22364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:46.741365 22364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:46.741672 22364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:46.761027 22364 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38213
I20260620 02:57:46.761075 22430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38213 every 8 connection(s)
I20260620 02:57:46.762488 22364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:57:46.762899 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 22364
I20260620 02:57:46.763123 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:57:46.763695 388 external_mini_cluster.cc:1428] Setting key b338f500239b7363cc603e427b632360
I20260620 02:57:46.767299 22431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:46.771783 22434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:46.765573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38073 (local address 127.0.97.62:38213)
0620 02:57:46.766471 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:46.766492 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:46.766537 (+ 45us) server_negotiation.cc:407] Connection header received
0620 02:57:46.768316 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:46.768338 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:46.768695 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:46.769097 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:46.770170 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:46.770176 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:46.770189 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:46.770202 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:46.770286 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:46.770305 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:46.770560 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:46.770692 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 02:57:46.770884 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":385,"threads_started":1}
I20260620 02:57:46.775537 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:57:46.779057 22431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e7bb86e7b2542eb9070e0ba1cbe4fdd: Bootstrap starting.
I20260620 02:57:46.783284 22434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:46.780219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38267 (local address 127.0.97.62:38213)
0620 02:57:46.780438 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:46.780443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:46.780890 (+ 447us) server_negotiation.cc:407] Connection header received
0620 02:57:46.781117 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:46.781122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:46.781179 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:46.781274 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:46.782485 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:46.782489 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:46.782493 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:46.782508 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:46.782553 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:46.782570 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:46.782677 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:46.782872 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 02:57:46.782923 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 02:57:46.783541 22431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e7bb86e7b2542eb9070e0ba1cbe4fdd: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:46.784834 22431 log.cc:824] T 00000000000000000000000000000000 P 2e7bb86e7b2542eb9070e0ba1cbe4fdd: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:46.788173 22431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e7bb86e7b2542eb9070e0ba1cbe4fdd: No bootstrap required, opened a new log
I20260620 02:57:46.796499 22431 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e7bb86e7b2542eb9070e0ba1cbe4fdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7bb86e7b2542eb9070e0ba1cbe4fdd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38213 } }
I20260620 02:57:46.796909 22431 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e7bb86e7b2542eb9070e0ba1cbe4fdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:46.796983 22431 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e7bb86e7b2542eb9070e0ba1cbe4fdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e7bb86e7b2542eb9070e0ba1cbe4fdd, State: Initialized, Role: FOLLOWER
I20260620 02:57:46.797637 22431 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e7bb86e7b2542eb9070e0ba1cbe4fdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7bb86e7b2542eb9070e0ba1cbe4fdd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38213 } }
I20260620 02:57:46.797905 22431 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e7bb86e7b2542eb9070e0ba1cbe4fdd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:57:46.798027 22431 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e7bb86e7b2542eb9070e0ba1cbe4fdd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:57:46.798173 22431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e7bb86e7b2542eb9070e0ba1cbe4fdd [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:46.801004 22431 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e7bb86e7b2542eb9070e0ba1cbe4fdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7bb86e7b2542eb9070e0ba1cbe4fdd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38213 } }
I20260620 02:57:46.801513 22431 leader_election.cc:302] T 00000000000000000000000000000000 P 2e7bb86e7b2542eb9070e0ba1cbe4fdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e7bb86e7b2542eb9070e0ba1cbe4fdd; no voters:
I20260620 02:57:46.802099 22431 leader_election.cc:288] T 00000000000000000000000000000000 P 2e7bb86e7b2542eb9070e0ba1cbe4fdd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:57:46.802251 22441 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e7bb86e7b2542eb9070e0ba1cbe4fdd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:57:46.803066 22441 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e7bb86e7b2542eb9070e0ba1cbe4fdd [term 1 LEADER]: Becoming Leader. State: Replica: 2e7bb86e7b2542eb9070e0ba1cbe4fdd, State: Running, Role: LEADER
I20260620 02:57:46.803635 22441 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e7bb86e7b2542eb9070e0ba1cbe4fdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7bb86e7b2542eb9070e0ba1cbe4fdd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38213 } }
I20260620 02:57:46.804592 22431 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e7bb86e7b2542eb9070e0ba1cbe4fdd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:57:46.807322 22442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e7bb86e7b2542eb9070e0ba1cbe4fdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e7bb86e7b2542eb9070e0ba1cbe4fdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7bb86e7b2542eb9070e0ba1cbe4fdd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38213 } } }
I20260620 02:57:46.807148 22443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e7bb86e7b2542eb9070e0ba1cbe4fdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e7bb86e7b2542eb9070e0ba1cbe4fdd. Latest consensus state: current_term: 1 leader_uuid: "2e7bb86e7b2542eb9070e0ba1cbe4fdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7bb86e7b2542eb9070e0ba1cbe4fdd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38213 } } }
I20260620 02:57:46.807714 22442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e7bb86e7b2542eb9070e0ba1cbe4fdd [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:46.807775 22443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e7bb86e7b2542eb9070e0ba1cbe4fdd [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:46.809381 22449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:57:46.818126 22449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:57:46.827260 22449 catalog_manager.cc:1347] Generated new cluster ID: a956af88a3c3406eb16550c6c110b797
I20260620 02:57:46.827390 22449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:57:46.845216 22449 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:57:46.845788 22449 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:57:46.853108 22449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e7bb86e7b2542eb9070e0ba1cbe4fdd: Generated new TSK 0
I20260620 02:57:46.854018 22449 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:57:46.859818 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 22364
W20260620 02:57:46.861315 22426 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38073
W20260620 02:57:46.862187 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 22364: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:57:46.862425 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 22364
2026-06-20T02:57:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1914 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:57:46 dist-test-slave-8p8c krb5kdc[22462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:57:46 dist-test-slave-8p8c krb5kdc[22462](info): set up 2 sockets
Jun 20 02:57:46 dist-test-slave-8p8c krb5kdc[22462](info): commencing operation
krb5kdc: starting...
W20260620 02:57:48.015041 388 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:57:48 dist-test-slave-8p8c krb5kdc[22462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:57:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:57:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:57:48.302181 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37051
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:57:48.651897 22475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:48.652418 22475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:48.652529 22475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:48.660167 22475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:57:48.660295 22475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:48.660343 22475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:48.660396 22475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:57:48.660441 22475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:57:48.670753 22475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:48.672050 22475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:48.673368 22475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:48.685876 22483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:48.685876 22481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:48.686138 22480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:48.686992 22475 server_base.cc:1034] running on GCE node
I20260620 02:57:48.687861 22475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:48.689316 22475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:48.690546 22475 hybrid_clock.cc:648] HybridClock initialized: now 1781924268690479 us; error 52 us; skew 500 ppm
I20260620 02:57:48.690927 22475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:57:48 dist-test-slave-8p8c krb5kdc[22462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:57:48.701633 22475 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:57:48.703390 22475 webserver.cc:460] Webserver started at http://127.0.97.62:40851/ using document root <none> and password file <none>
I20260620 02:57:48.704133 22475 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:48.704290 22475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:48.704617 22475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:48.707762 22475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "4a07f7d65fcd4978b7d432d74fc59f91"
format_stamp: "Formatted at 2026-06-20 02:57:48 on dist-test-slave-8p8c"
server_key: "d5bac1e346b9daf257abf359090e4d4b"
server_key_iv: "e4b8d877027b9a299a5a32b4325f060b"
server_key_version: "encryptionkey@0"
I20260620 02:57:48.708650 22475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "4a07f7d65fcd4978b7d432d74fc59f91"
format_stamp: "Formatted at 2026-06-20 02:57:48 on dist-test-slave-8p8c"
server_key: "d5bac1e346b9daf257abf359090e4d4b"
server_key_iv: "e4b8d877027b9a299a5a32b4325f060b"
server_key_version: "encryptionkey@0"
I20260620 02:57:48.713995 22475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:57:48.718348 22490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:48.719753 22475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260620 02:57:48.719947 22475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "4a07f7d65fcd4978b7d432d74fc59f91"
format_stamp: "Formatted at 2026-06-20 02:57:48 on dist-test-slave-8p8c"
server_key: "d5bac1e346b9daf257abf359090e4d4b"
server_key_iv: "e4b8d877027b9a299a5a32b4325f060b"
server_key_version: "encryptionkey@0"
I20260620 02:57:48.720187 22475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:48.749138 22475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:48.753409 22475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:48.753706 22475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:48.773408 22475 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37051
I20260620 02:57:48.773440 22541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37051 every 8 connection(s)
I20260620 02:57:48.774868 22475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:57:48.778820 22542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:48.779628 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 22475
I20260620 02:57:48.779810 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:57:48.780409 388 external_mini_cluster.cc:1428] Setting key ff90ebc96c93f0d87d81d97323246761
I20260620 02:57:48.793895 22542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a07f7d65fcd4978b7d432d74fc59f91: Bootstrap starting.
Jun 20 02:57:48 dist-test-slave-8p8c krb5kdc[22462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:57:48.798023 22542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a07f7d65fcd4978b7d432d74fc59f91: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:48.799222 22542 log.cc:824] T 00000000000000000000000000000000 P 4a07f7d65fcd4978b7d432d74fc59f91: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:48.802178 22542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a07f7d65fcd4978b7d432d74fc59f91: No bootstrap required, opened a new log
I20260620 02:57:48.805720 22545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:48.782684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34415 (local address 127.0.97.62:37051)
0620 02:57:48.783672 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:48.783688 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:48.783740 (+ 52us) server_negotiation.cc:407] Connection header received
0620 02:57:48.785658 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:48.785690 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:48.786222 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:48.786733 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:48.788535 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:48.789877 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:48.791853 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:48.792730 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:48.792898 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:48.798249 (+ 5351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:48.798259 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:48.798275 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:48.798335 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:48.801572 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:48.802311 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:48.802324 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:48.802330 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:48.802549 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:48.802956 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:48.802967 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:48.802970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:48.803499 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:48.803853 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:48.804281 (+ 428us) server_negotiation.cc:299] Negotiation successful
0620 02:57:48.804715 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":457,"threads_started":1}
I20260620 02:57:48.809674 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:57:48.810819 22542 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a07f7d65fcd4978b7d432d74fc59f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a07f7d65fcd4978b7d432d74fc59f91" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37051 } }
I20260620 02:57:48.811197 22542 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a07f7d65fcd4978b7d432d74fc59f91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:48.811303 22542 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a07f7d65fcd4978b7d432d74fc59f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a07f7d65fcd4978b7d432d74fc59f91, State: Initialized, Role: FOLLOWER
I20260620 02:57:48.811861 22542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a07f7d65fcd4978b7d432d74fc59f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a07f7d65fcd4978b7d432d74fc59f91" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37051 } }
I20260620 02:57:48.812110 22542 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a07f7d65fcd4978b7d432d74fc59f91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:57:48.812247 22542 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a07f7d65fcd4978b7d432d74fc59f91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:57:48.812412 22542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a07f7d65fcd4978b7d432d74fc59f91 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:48.815186 22542 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a07f7d65fcd4978b7d432d74fc59f91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a07f7d65fcd4978b7d432d74fc59f91" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37051 } }
I20260620 02:57:48.815701 22542 leader_election.cc:302] T 00000000000000000000000000000000 P 4a07f7d65fcd4978b7d432d74fc59f91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a07f7d65fcd4978b7d432d74fc59f91; no voters:
I20260620 02:57:48.816272 22542 leader_election.cc:288] T 00000000000000000000000000000000 P 4a07f7d65fcd4978b7d432d74fc59f91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:57:48.816437 22552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a07f7d65fcd4978b7d432d74fc59f91 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:57:48.817080 22552 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a07f7d65fcd4978b7d432d74fc59f91 [term 1 LEADER]: Becoming Leader. State: Replica: 4a07f7d65fcd4978b7d432d74fc59f91, State: Running, Role: LEADER
I20260620 02:57:48.817718 22552 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a07f7d65fcd4978b7d432d74fc59f91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a07f7d65fcd4978b7d432d74fc59f91" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37051 } }
I20260620 02:57:48.820570 22542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a07f7d65fcd4978b7d432d74fc59f91 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:57:48.820451 22554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a07f7d65fcd4978b7d432d74fc59f91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a07f7d65fcd4978b7d432d74fc59f91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a07f7d65fcd4978b7d432d74fc59f91" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37051 } } }
I20260620 02:57:48.820870 22554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a07f7d65fcd4978b7d432d74fc59f91 [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:48.821406 22553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a07f7d65fcd4978b7d432d74fc59f91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a07f7d65fcd4978b7d432d74fc59f91. Latest consensus state: current_term: 1 leader_uuid: "4a07f7d65fcd4978b7d432d74fc59f91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a07f7d65fcd4978b7d432d74fc59f91" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37051 } } }
I20260620 02:57:48.821709 22553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a07f7d65fcd4978b7d432d74fc59f91 [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:48.830333 22556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:57:48.833283 22545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:48.814088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47275 (local address 127.0.97.62:37051)
0620 02:57:48.814268 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:48.814276 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:48.814772 (+ 496us) server_negotiation.cc:407] Connection header received
0620 02:57:48.814889 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:48.814895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:48.814978 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:48.815087 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:48.817077 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:48.818197 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:48.820054 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:48.820852 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:48.820959 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:48.824955 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:48.824965 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:48.824973 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:48.825035 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:48.829704 (+ 4669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:48.830629 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:48.830633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:48.830635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:48.830744 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:48.831153 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:48.831158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:48.831162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:48.831791 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:48.832622 (+ 831us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:48.832780 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 02:57:48.833001 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 02:57:48.837512 22556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:57:48.847035 22556 catalog_manager.cc:1347] Generated new cluster ID: b6a86bbf07f048779414d1c344c4d2e5
I20260620 02:57:48.847142 22556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:57:48.862591 22556 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:57:48.863904 22556 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:57:48.874612 22556 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a07f7d65fcd4978b7d432d74fc59f91: Generated new TSK 0
I20260620 02:57:48.875752 22556 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:57:48.883505 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 22475
I20260620 02:57:48.884884 22537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:57:48.983247 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 22475
2026-06-20T02:57:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2123 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:57:49 dist-test-slave-8p8c krb5kdc[22575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:57:49 dist-test-slave-8p8c krb5kdc[22575](info): set up 2 sockets
Jun 20 02:57:49 dist-test-slave-8p8c krb5kdc[22575](info): commencing operation
krb5kdc: starting...
W20260620 02:57:50.114979 388 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:57:50 dist-test-slave-8p8c krb5kdc[22575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:57:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:57:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:57:50.383087 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45331
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:57:50.725945 22588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:50.726318 22588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:50.726536 22588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:50.733840 22588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:57:50.734014 22588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:50.734117 22588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:50.734225 22588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:57:50.734304 22588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:57:50.744645 22588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:50.746059 22588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:50.747537 22588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:50.760066 22594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:50.760102 22596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:50.760126 22593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:50.760777 22588 server_base.cc:1034] running on GCE node
I20260620 02:57:50.761613 22588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:50.763156 22588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:50.764385 22588 hybrid_clock.cc:648] HybridClock initialized: now 1781924270764327 us; error 42 us; skew 500 ppm
I20260620 02:57:50.764770 22588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:57:50 dist-test-slave-8p8c krb5kdc[22575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:57:50.768164 22588 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:57:50.769587 22588 webserver.cc:460] Webserver started at http://127.0.97.62:34125/ using document root <none> and password file <none>
I20260620 02:57:50.770237 22588 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:50.770336 22588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:50.770648 22588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:50.773620 22588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "b090ba4fdc5e41f588ec16571891a755"
format_stamp: "Formatted at 2026-06-20 02:57:50 on dist-test-slave-8p8c"
server_key: "9fc94c45c28e9737b48cf44f8933420f"
server_key_iv: "4afe8e70332007d117bf7c2092ff9e17"
server_key_version: "encryptionkey@0"
I20260620 02:57:50.774322 22588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "b090ba4fdc5e41f588ec16571891a755"
format_stamp: "Formatted at 2026-06-20 02:57:50 on dist-test-slave-8p8c"
server_key: "9fc94c45c28e9737b48cf44f8933420f"
server_key_iv: "4afe8e70332007d117bf7c2092ff9e17"
server_key_version: "encryptionkey@0"
I20260620 02:57:50.779528 22588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 02:57:50.782995 22603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:50.784557 22588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:57:50.784756 22588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "b090ba4fdc5e41f588ec16571891a755"
format_stamp: "Formatted at 2026-06-20 02:57:50 on dist-test-slave-8p8c"
server_key: "9fc94c45c28e9737b48cf44f8933420f"
server_key_iv: "4afe8e70332007d117bf7c2092ff9e17"
server_key_version: "encryptionkey@0"
I20260620 02:57:50.784938 22588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:50.799240 22588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:50.803972 22588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:50.804275 22588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:50.823983 22588 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45331
I20260620 02:57:50.824060 22654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45331 every 8 connection(s)
I20260620 02:57:50.825419 22588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:57:50.829129 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 22588
I20260620 02:57:50.829324 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:57:50.829316 22655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:50.829777 388 external_mini_cluster.cc:1428] Setting key b5e3666fe8a4bd1d9ea6de65a3196825
I20260620 02:57:50.843489 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:50.831516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36981 (local address 127.0.97.62:45331)
0620 02:57:50.832363 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:50.832375 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:50.832416 (+ 41us) server_negotiation.cc:407] Connection header received
0620 02:57:50.834021 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:50.834043 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:50.834539 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:50.834876 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:50.835464 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:50.836797 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:50.838641 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:50.839329 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:50.839465 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:50.840782 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:50.840790 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:50.840805 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:50.841493 (+ 688us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:50.841574 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:50.841607 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:50.841710 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:50.841995 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 02:57:50.842302 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":395,"threads_started":1}
I20260620 02:57:50.843487 22655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b090ba4fdc5e41f588ec16571891a755: Bootstrap starting.
I20260620 02:57:50.847213 22655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b090ba4fdc5e41f588ec16571891a755: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:50.847699 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:57:50.848592 22655 log.cc:824] T 00000000000000000000000000000000 P b090ba4fdc5e41f588ec16571891a755: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:50.851578 22655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b090ba4fdc5e41f588ec16571891a755: No bootstrap required, opened a new log
I20260620 02:57:50.859180 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:50.852082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40201 (local address 127.0.97.62:45331)
0620 02:57:50.852282 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:50.852289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:50.852629 (+ 340us) server_negotiation.cc:407] Connection header received
0620 02:57:50.852781 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:50.852786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:50.852866 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:50.852986 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:50.853609 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:50.854801 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:50.856272 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:50.857032 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:50.857165 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:50.858285 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:50.858291 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:50.858296 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:50.858319 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:50.858403 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:50.858425 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:50.858522 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:50.858744 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 02:57:50.858840 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 02:57:50.861256 22655 raft_consensus.cc:348] T 00000000000000000000000000000000 P b090ba4fdc5e41f588ec16571891a755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b090ba4fdc5e41f588ec16571891a755" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45331 } }
I20260620 02:57:50.861653 22655 raft_consensus.cc:374] T 00000000000000000000000000000000 P b090ba4fdc5e41f588ec16571891a755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:50.861776 22655 raft_consensus.cc:729] T 00000000000000000000000000000000 P b090ba4fdc5e41f588ec16571891a755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b090ba4fdc5e41f588ec16571891a755, State: Initialized, Role: FOLLOWER
I20260620 02:57:50.862419 22655 consensus_queue.cc:260] T 00000000000000000000000000000000 P b090ba4fdc5e41f588ec16571891a755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b090ba4fdc5e41f588ec16571891a755" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45331 } }
I20260620 02:57:50.862670 22655 raft_consensus.cc:388] T 00000000000000000000000000000000 P b090ba4fdc5e41f588ec16571891a755 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:57:50.862828 22655 raft_consensus.cc:482] T 00000000000000000000000000000000 P b090ba4fdc5e41f588ec16571891a755 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:57:50.862989 22655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b090ba4fdc5e41f588ec16571891a755 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:50.865165 22655 raft_consensus.cc:504] T 00000000000000000000000000000000 P b090ba4fdc5e41f588ec16571891a755 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b090ba4fdc5e41f588ec16571891a755" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45331 } }
I20260620 02:57:50.865715 22655 leader_election.cc:302] T 00000000000000000000000000000000 P b090ba4fdc5e41f588ec16571891a755 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b090ba4fdc5e41f588ec16571891a755; no voters:
I20260620 02:57:50.866544 22655 leader_election.cc:288] T 00000000000000000000000000000000 P b090ba4fdc5e41f588ec16571891a755 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:57:50.866734 22665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b090ba4fdc5e41f588ec16571891a755 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:57:50.867568 22665 raft_consensus.cc:686] T 00000000000000000000000000000000 P b090ba4fdc5e41f588ec16571891a755 [term 1 LEADER]: Becoming Leader. State: Replica: b090ba4fdc5e41f588ec16571891a755, State: Running, Role: LEADER
I20260620 02:57:50.868178 22665 consensus_queue.cc:237] T 00000000000000000000000000000000 P b090ba4fdc5e41f588ec16571891a755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b090ba4fdc5e41f588ec16571891a755" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45331 } }
I20260620 02:57:50.868839 22655 sys_catalog.cc:564] T 00000000000000000000000000000000 P b090ba4fdc5e41f588ec16571891a755 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:57:50.871867 22667 sys_catalog.cc:455] T 00000000000000000000000000000000 P b090ba4fdc5e41f588ec16571891a755 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b090ba4fdc5e41f588ec16571891a755. Latest consensus state: current_term: 1 leader_uuid: "b090ba4fdc5e41f588ec16571891a755" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b090ba4fdc5e41f588ec16571891a755" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45331 } } }
I20260620 02:57:50.872273 22666 sys_catalog.cc:455] T 00000000000000000000000000000000 P b090ba4fdc5e41f588ec16571891a755 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b090ba4fdc5e41f588ec16571891a755" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b090ba4fdc5e41f588ec16571891a755" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45331 } } }
I20260620 02:57:50.872624 22666 sys_catalog.cc:458] T 00000000000000000000000000000000 P b090ba4fdc5e41f588ec16571891a755 [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:50.872624 22667 sys_catalog.cc:458] T 00000000000000000000000000000000 P b090ba4fdc5e41f588ec16571891a755 [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:50.875017 22671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:57:50.881131 22671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:57:50.891358 22671 catalog_manager.cc:1347] Generated new cluster ID: e0d0d463a6bb4811b93323108a9af6f5
I20260620 02:57:50.891510 22671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:57:50.906136 22671 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:57:50.907531 22671 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:57:50.915084 22671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b090ba4fdc5e41f588ec16571891a755: Generated new TSK 0
I20260620 02:57:50.916013 22671 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:57:50.927433 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 22588
W20260620 02:57:50.929173 22650 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36981
W20260620 02:57:50.929869 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 22588: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:57:50.930028 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 22588
2026-06-20T02:57:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1945 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:57:51 dist-test-slave-8p8c krb5kdc[22686](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:57:51 dist-test-slave-8p8c krb5kdc[22686](info): set up 2 sockets
Jun 20 02:57:51 dist-test-slave-8p8c krb5kdc[22686](info): commencing operation
krb5kdc: starting...
W20260620 02:57:52.083214 388 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:57:52 dist-test-slave-8p8c krb5kdc[22686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:57:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:57:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:57:52.353374 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46109
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:57:52.695487 22699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:52.695829 22699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:52.696002 22699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:52.703162 22699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:57:52.703315 22699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:52.703410 22699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:52.703518 22699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:57:52.703593 22699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:57:52.713758 22699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:52.715134 22699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:52.716528 22699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:52.728473 22707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:52.728619 22705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:52.728592 22704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:52.729803 22699 server_base.cc:1034] running on GCE node
I20260620 02:57:52.730652 22699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:52.732024 22699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:52.733289 22699 hybrid_clock.cc:648] HybridClock initialized: now 1781924272733240 us; error 33 us; skew 500 ppm
I20260620 02:57:52.733685 22699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:57:52 dist-test-slave-8p8c krb5kdc[22686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:57:52.736999 22699 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:57:52.738663 22699 webserver.cc:460] Webserver started at http://127.0.97.62:35089/ using document root <none> and password file <none>
I20260620 02:57:52.739354 22699 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:52.739495 22699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:52.739801 22699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:52.742900 22699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "b1a8ca44c1c84c3985cd14cd7ccaf59e"
format_stamp: "Formatted at 2026-06-20 02:57:52 on dist-test-slave-8p8c"
server_key: "e2da136e06b7e3f0b293141932456364"
server_key_iv: "ec4b71a0f930b453e92a1e83fdf70cd4"
server_key_version: "encryptionkey@0"
I20260620 02:57:52.743681 22699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "b1a8ca44c1c84c3985cd14cd7ccaf59e"
format_stamp: "Formatted at 2026-06-20 02:57:52 on dist-test-slave-8p8c"
server_key: "e2da136e06b7e3f0b293141932456364"
server_key_iv: "ec4b71a0f930b453e92a1e83fdf70cd4"
server_key_version: "encryptionkey@0"
I20260620 02:57:52.748967 22699 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260620 02:57:52.752887 22714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:52.754566 22699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:57:52.754747 22699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "b1a8ca44c1c84c3985cd14cd7ccaf59e"
format_stamp: "Formatted at 2026-06-20 02:57:52 on dist-test-slave-8p8c"
server_key: "e2da136e06b7e3f0b293141932456364"
server_key_iv: "ec4b71a0f930b453e92a1e83fdf70cd4"
server_key_version: "encryptionkey@0"
I20260620 02:57:52.754953 22699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:52.760859 22699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:52.765465 22699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:52.765772 22699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:52.785585 22699 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46109
I20260620 02:57:52.785635 22765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46109 every 8 connection(s)
I20260620 02:57:52.787149 22699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:57:52.791049 22766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:52.792032 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 22699
I20260620 02:57:52.792212 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:57:52.792758 388 external_mini_cluster.cc:1428] Setting key c8f039442c9dc9da98b93e33186f494e
I20260620 02:57:52.801607 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:52.794801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53435 (local address 127.0.97.62:46109)
0620 02:57:52.796226 (+ 1425us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:52.796238 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:52.796284 (+ 46us) server_negotiation.cc:407] Connection header received
0620 02:57:52.797910 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:52.797936 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:52.798305 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:52.798723 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:52.799856 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:52.799863 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:52.799876 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:52.799892 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:52.800063 (+ 171us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:52.800098 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:52.800277 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:52.800432 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 02:57:52.800646 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1142,"thread_start_us":876,"threads_started":1}
I20260620 02:57:52.805157 22766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1a8ca44c1c84c3985cd14cd7ccaf59e: Bootstrap starting.
I20260620 02:57:52.805442 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:57:52.808403 22766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1a8ca44c1c84c3985cd14cd7ccaf59e: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:52.809465 22766 log.cc:824] T 00000000000000000000000000000000 P b1a8ca44c1c84c3985cd14cd7ccaf59e: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:52.813138 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:52.810018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57795 (local address 127.0.97.62:46109)
0620 02:57:52.810227 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:52.810235 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:52.810758 (+ 523us) server_negotiation.cc:407] Connection header received
0620 02:57:52.810966 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:52.810971 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:52.811029 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:52.811151 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:52.812270 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:52.812279 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:52.812287 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 02:57:52.812307 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:52.812368 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 02:57:52.812390 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:52.812507 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:52.812692 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 02:57:52.812783 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 02:57:52.813140 22766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1a8ca44c1c84c3985cd14cd7ccaf59e: No bootstrap required, opened a new log
I20260620 02:57:52.821260 22766 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1a8ca44c1c84c3985cd14cd7ccaf59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a8ca44c1c84c3985cd14cd7ccaf59e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46109 } }
I20260620 02:57:52.821635 22766 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1a8ca44c1c84c3985cd14cd7ccaf59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 02:57:52.821709 22766 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1a8ca44c1c84c3985cd14cd7ccaf59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1a8ca44c1c84c3985cd14cd7ccaf59e, State: Initialized, Role: FOLLOWER
I20260620 02:57:52.822237 22766 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1a8ca44c1c84c3985cd14cd7ccaf59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a8ca44c1c84c3985cd14cd7ccaf59e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46109 } }
I20260620 02:57:52.822513 22766 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1a8ca44c1c84c3985cd14cd7ccaf59e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 02:57:52.822626 22766 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1a8ca44c1c84c3985cd14cd7ccaf59e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 02:57:52.822746 22766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1a8ca44c1c84c3985cd14cd7ccaf59e [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:52.825263 22766 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1a8ca44c1c84c3985cd14cd7ccaf59e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a8ca44c1c84c3985cd14cd7ccaf59e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46109 } }
I20260620 02:57:52.825789 22766 leader_election.cc:302] T 00000000000000000000000000000000 P b1a8ca44c1c84c3985cd14cd7ccaf59e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1a8ca44c1c84c3985cd14cd7ccaf59e; no voters:
I20260620 02:57:52.826333 22766 leader_election.cc:288] T 00000000000000000000000000000000 P b1a8ca44c1c84c3985cd14cd7ccaf59e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 02:57:52.826531 22776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1a8ca44c1c84c3985cd14cd7ccaf59e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:57:52.827229 22776 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1a8ca44c1c84c3985cd14cd7ccaf59e [term 1 LEADER]: Becoming Leader. State: Replica: b1a8ca44c1c84c3985cd14cd7ccaf59e, State: Running, Role: LEADER
I20260620 02:57:52.827766 22776 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1a8ca44c1c84c3985cd14cd7ccaf59e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a8ca44c1c84c3985cd14cd7ccaf59e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46109 } }
I20260620 02:57:52.828483 22766 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1a8ca44c1c84c3985cd14cd7ccaf59e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:57:52.830605 22777 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1a8ca44c1c84c3985cd14cd7ccaf59e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1a8ca44c1c84c3985cd14cd7ccaf59e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a8ca44c1c84c3985cd14cd7ccaf59e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46109 } } }
I20260620 02:57:52.831027 22777 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1a8ca44c1c84c3985cd14cd7ccaf59e [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:52.831023 22778 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1a8ca44c1c84c3985cd14cd7ccaf59e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1a8ca44c1c84c3985cd14cd7ccaf59e. Latest consensus state: current_term: 1 leader_uuid: "b1a8ca44c1c84c3985cd14cd7ccaf59e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a8ca44c1c84c3985cd14cd7ccaf59e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46109 } } }
I20260620 02:57:52.833217 22778 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1a8ca44c1c84c3985cd14cd7ccaf59e [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:52.834620 22784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:57:52.839133 22784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:57:52.848676 22784 catalog_manager.cc:1347] Generated new cluster ID: 5d6dded5d46a49cfa16d323bf5c46792
I20260620 02:57:52.848816 22784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:57:52.862561 22784 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:57:52.863099 22784 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:57:52.873855 22784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1a8ca44c1c84c3985cd14cd7ccaf59e: Generated new TSK 0
I20260620 02:57:52.874959 22784 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:57:52.890481 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 22699
W20260620 02:57:52.891813 22761 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53435
W20260620 02:57:52.892599 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 22699: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 02:57:52.892755 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 22699
2026-06-20T02:57:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1962 ms)
[----------] 18 tests from AuthTokenIssuingTest (35713 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:57:53 dist-test-slave-8p8c krb5kdc[22797](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:57:53 dist-test-slave-8p8c krb5kdc[22797](info): set up 2 sockets
Jun 20 02:57:53 dist-test-slave-8p8c krb5kdc[22797](info): commencing operation
krb5kdc: starting...
W20260620 02:57:54.027100 388 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.009s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:57:54 dist-test-slave-8p8c krb5kdc[22797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:57:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:57:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:57:54.298803 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36041
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36041,127.0.97.61:40233,127.0.97.60:42507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:57:54.631574 22810 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 02:57:54.631899 22810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:54.631996 22810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:54.632133 22810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:54.639592 22810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:57:54.639700 22810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:54.639760 22810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:54.639803 22810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:57:54.640064 22810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:57:54.650211 22810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36041,127.0.97.61:40233,127.0.97.60:42507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:54.651520 22810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:54.652969 22810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:54.665233 22815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:54.665272 22816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:54.665315 22818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:54.666327 22810 server_base.cc:1034] running on GCE node
I20260620 02:57:54.667232 22810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:54.668637 22810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:54.669939 22810 hybrid_clock.cc:648] HybridClock initialized: now 1781924274669878 us; error 34 us; skew 500 ppm
I20260620 02:57:54.670356 22810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:57:54 dist-test-slave-8p8c krb5kdc[22797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:57:54.677966 22810 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:57:54.679616 22810 webserver.cc:460] Webserver started at http://127.0.97.62:43215/ using document root <none> and password file <none>
I20260620 02:57:54.680313 22810 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:54.680401 22810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:54.680656 22810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:54.683663 22810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "5e24609ff86d44a6aeea289d139fbf53"
format_stamp: "Formatted at 2026-06-20 02:57:54 on dist-test-slave-8p8c"
server_key: "88fdb2e22a9f55e58a82921aa557f10a"
server_key_iv: "d38f554917b24b705c3679fba4385e06"
server_key_version: "encryptionkey@0"
I20260620 02:57:54.684386 22810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "5e24609ff86d44a6aeea289d139fbf53"
format_stamp: "Formatted at 2026-06-20 02:57:54 on dist-test-slave-8p8c"
server_key: "88fdb2e22a9f55e58a82921aa557f10a"
server_key_iv: "d38f554917b24b705c3679fba4385e06"
server_key_version: "encryptionkey@0"
I20260620 02:57:54.689450 22810 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 02:57:54.692956 22825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:54.694490 22810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 02:57:54.694657 22810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "5e24609ff86d44a6aeea289d139fbf53"
format_stamp: "Formatted at 2026-06-20 02:57:54 on dist-test-slave-8p8c"
server_key: "88fdb2e22a9f55e58a82921aa557f10a"
server_key_iv: "d38f554917b24b705c3679fba4385e06"
server_key_version: "encryptionkey@0"
I20260620 02:57:54.694857 22810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:54.714871 22810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:54.719358 22810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:54.719635 22810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:54.738535 22810 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36041
I20260620 02:57:54.738643 22876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36041 every 8 connection(s)
I20260620 02:57:54.739876 22810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:57:54.743664 22877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:54.744270 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 22810
I20260620 02:57:54.744499 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:57:54.745129 388 external_mini_cluster.cc:1428] Setting key a2d798c800b57fcfa0a8b8308f7ddb20
I20260620 02:57:54.750567 22877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36041 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:57:54 dist-test-slave-8p8c krb5kdc[22797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260620 02:57:54.775532 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:54.753704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36041 (local address 127.0.0.1:49964)
0620 02:57:54.754556 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:54.754608 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:54.756629 (+ 2021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:54.757633 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:54.757638 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:54.758234 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:57:54.758688 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:54.758708 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:54.761011 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:54.761015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:54.762139 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:54.762152 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:54.763562 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:54.763567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:54.763844 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:54.764605 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:57:54.764629 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:57:54.767246 (+ 2617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:57:54.771186 (+ 3940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:57:54.771196 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:57:54.771205 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:57:54.771582 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:57:54.771979 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:57:54.771982 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:57:54.771984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:57:54.772177 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:57:54.772984 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:57:54.772992 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:57:54.773219 (+ 227us) client_negotiation.cc:769] Sending connection context
0620 02:57:54.773472 (+ 253us) client_negotiation.cc:240] Negotiation successful
0620 02:57:54.774081 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":232,"thread_start_us":360,"threads_started":1}
I20260620 02:57:54.775534 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:54.755022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49964 (local address 127.0.97.62:36041)
0620 02:57:54.755709 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:54.755720 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:54.755774 (+ 54us) server_negotiation.cc:407] Connection header received
0620 02:57:54.757014 (+ 1240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:54.757024 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:54.757349 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:54.757571 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:54.758922 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:54.760823 (+ 1901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:54.762293 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:54.763315 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:54.763505 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:54.767503 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:54.767517 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:54.767535 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:54.767577 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:54.771063 (+ 3486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:54.771718 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:54.771725 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:54.771727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:54.771878 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:54.772322 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:54.772327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:54.772331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:54.772828 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:54.773084 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:54.773593 (+ 509us) server_negotiation.cc:299] Negotiation successful
0620 02:57:54.773968 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":462,"thread_start_us":311,"threads_started":1}
I20260620 02:57:54.778828 22877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:57:54.779913 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:54.779517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40233 (local address 127.0.0.1:40620)
0620 02:57:54.779688 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:54.779807 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 02:57:54.780252 22829 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40233: connect: Connection refused (error 111)
I20260620 02:57:54.781831 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:54.781483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40233 (local address 127.0.0.1:40634)
0620 02:57:54.781629 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:54.781709 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 02:57:54.782049 22877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40233: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40233: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260620 02:57:54.802151 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40233
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36041,127.0.97.61:40233,127.0.97.60:42507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:57:54.831368 22877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } attempt: 1
I20260620 02:57:54.832705 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:54.832150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40233 (local address 127.0.0.1:40646)
0620 02:57:54.832391 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:54.832525 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 02:57:54.833952 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:54.833529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40233 (local address 127.0.0.1:40648)
0620 02:57:54.833697 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:54.833785 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 02:57:54.834249 22877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40233: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40233: connect: Connection refused (error 111)
I20260620 02:57:54.902526 22877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } attempt: 2
I20260620 02:57:54.903813 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:54.903240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40233 (local address 127.0.0.1:40654)
0620 02:57:54.903513 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:54.903640 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 02:57:54.904937 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:54.904513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40233 (local address 127.0.0.1:40666)
0620 02:57:54.904687 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:54.904775 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 02:57:54.905184 22877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40233: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40233: connect: Connection refused (error 111)
I20260620 02:57:54.996521 22877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } attempt: 3
I20260620 02:57:54.998003 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:54.997358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40233 (local address 127.0.0.1:40680)
0620 02:57:54.997646 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:54.997786 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 02:57:54.999576 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:54.999017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40233 (local address 127.0.0.1:40692)
0620 02:57:54.999227 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:54.999345 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 02:57:54.999888 22877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40233: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40233: connect: Connection refused (error 111)
I20260620 02:57:55.143148 22877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } attempt: 4
W20260620 02:57:55.144027 22884 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
I20260620 02:57:55.144474 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.143798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40233 (local address 127.0.0.1:40698)
0620 02:57:55.144169 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:55.144293 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
W20260620 02:57:55.144542 22884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:55.144657 22884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:55.144851 22884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 02:57:55.145792 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.145309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40233 (local address 127.0.0.1:40700)
0620 02:57:55.145502 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:55.145629 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 02:57:55.146055 22877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40233: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40233: connect: Connection refused (error 111)
W20260620 02:57:55.153362 22884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:57:55.153445 22884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:55.153555 22884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:55.153631 22884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:57:55.153736 22884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:57:55.163636 22884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36041,127.0.97.61:40233,127.0.97.60:42507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:55.164908 22884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:55.166236 22884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:55.178932 22889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:55.178891 22890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:55.178978 22892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:55.181088 22884 server_base.cc:1034] running on GCE node
I20260620 02:57:55.181922 22884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:55.183473 22884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:55.184741 22884 hybrid_clock.cc:648] HybridClock initialized: now 1781924275184677 us; error 40 us; skew 500 ppm
I20260620 02:57:55.185094 22884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:57:55 dist-test-slave-8p8c krb5kdc[22797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:57:55.188874 22884 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260620 02:57:55.190730 22884 webserver.cc:460] Webserver started at http://127.0.97.61:46717/ using document root <none> and password file <none>
I20260620 02:57:55.191453 22884 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:55.191596 22884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:55.191924 22884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:55.195154 22884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-1/data/instance:
uuid: "eb8a95ae2e9346319372e0078599ec5a"
format_stamp: "Formatted at 2026-06-20 02:57:55 on dist-test-slave-8p8c"
server_key: "106584cb5e3122a64da14a0b93b951ca"
server_key_iv: "30f59584ecc92b3c972bd70ecdd72101"
server_key_version: "encryptionkey@0"
I20260620 02:57:55.195951 22884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-1/wal/instance:
uuid: "eb8a95ae2e9346319372e0078599ec5a"
format_stamp: "Formatted at 2026-06-20 02:57:55 on dist-test-slave-8p8c"
server_key: "106584cb5e3122a64da14a0b93b951ca"
server_key_iv: "30f59584ecc92b3c972bd70ecdd72101"
server_key_version: "encryptionkey@0"
I20260620 02:57:55.201378 22884 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 02:57:55.204895 22899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:55.206288 22884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:57:55.206492 22884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-1/wal
uuid: "eb8a95ae2e9346319372e0078599ec5a"
format_stamp: "Formatted at 2026-06-20 02:57:55 on dist-test-slave-8p8c"
server_key: "106584cb5e3122a64da14a0b93b951ca"
server_key_iv: "30f59584ecc92b3c972bd70ecdd72101"
server_key_version: "encryptionkey@0"
I20260620 02:57:55.206740 22884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:55.217583 22884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:55.221658 22884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:55.221951 22884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:55.240074 22884 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:40233
I20260620 02:57:55.240103 22950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:40233 every 8 connection(s)
I20260620 02:57:55.241463 22884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-1/data/info.pb
I20260620 02:57:55.245191 22951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:55.247617 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 22884
I20260620 02:57:55.247820 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-1/wal/instance
I20260620 02:57:55.248260 388 external_mini_cluster.cc:1428] Setting key 3a4faee1741b088c678b6021b9937be0
I20260620 02:57:55.251752 22951 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36041 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:57:55 dist-test-slave-8p8c krb5kdc[22797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260620 02:57:55.273545 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.254426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49976 (local address 127.0.97.62:36041)
0620 02:57:55.254626 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:55.254630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:55.255921 (+ 1291us) server_negotiation.cc:407] Connection header received
0620 02:57:55.258302 (+ 2381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:55.258309 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:55.258449 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:55.258575 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:55.260565 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.261261 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.262921 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.263349 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.263489 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:55.267204 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:55.267208 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:55.267212 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:55.267243 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:55.270325 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:55.271165 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:55.271170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:55.271174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:55.271301 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:55.271854 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:55.271858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:55.271860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:55.272154 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:55.272267 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:55.273074 (+ 807us) server_negotiation.cc:299] Negotiation successful
0620 02:57:55.273278 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260620 02:57:55.274741 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.254974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36041 (local address 127.0.0.1:49976)
0620 02:57:55.255746 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:55.255792 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:55.258022 (+ 2230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:55.258837 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:55.258852 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:55.259721 (+ 869us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:57:55.260395 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:55.260412 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.261416 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.261421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:55.262769 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:55.262781 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.263575 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.263583 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:55.263815 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:55.264508 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:57:55.264531 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:57:55.266965 (+ 2434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:57:55.270571 (+ 3606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:57:55.270580 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:57:55.270597 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:57:55.271038 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:57:55.271469 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:57:55.271476 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:57:55.271479 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:57:55.271718 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:57:55.272338 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:57:55.272345 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:57:55.272725 (+ 380us) client_negotiation.cc:769] Sending connection context
0620 02:57:55.272957 (+ 232us) client_negotiation.cc:240] Negotiation successful
0620 02:57:55.273405 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":337,"threads_started":1}
I20260620 02:57:55.276892 22951 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:57:55 dist-test-slave-8p8c krb5kdc[22797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 02:57:55.294642 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.277798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40233 (local address 127.0.0.1:40704)
0620 02:57:55.278017 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:55.278043 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:55.278176 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:55.279126 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:55.279135 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:55.279719 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:57:55.279962 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:55.279979 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.281413 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.281430 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:55.283251 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:55.283271 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.284106 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.284117 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:55.284399 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:55.285103 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:57:55.285126 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:57:55.287558 (+ 2432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:57:55.292148 (+ 4590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:57:55.292152 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:57:55.292155 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:57:55.292524 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:57:55.292945 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:57:55.292948 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:57:55.292950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:57:55.293102 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:57:55.293854 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:57:55.293857 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:57:55.293970 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 02:57:55.294109 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 02:57:55.294309 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 02:57:55.295099 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.278000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40704 (local address 127.0.97.61:40233)
0620 02:57:55.278694 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:55.278700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:55.278724 (+ 24us) server_negotiation.cc:407] Connection header received
0620 02:57:55.278768 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:55.278773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:55.278948 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:55.279113 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:55.280167 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.281228 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.283408 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.283930 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.284031 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:55.287788 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:55.287805 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:55.287826 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:55.287868 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:55.291985 (+ 4117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:55.292660 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:55.292672 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:55.292678 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:55.292821 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:55.293219 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:55.293226 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:55.293233 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:55.293736 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:55.293944 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:55.294266 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 02:57:55.294577 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":418,"threads_started":1}
I20260620 02:57:55.297858 22951 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42507 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:57:55.298985 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.298532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42507 (local address 127.0.0.1:42230)
0620 02:57:55.298712 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:55.298842 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260620 02:57:55.299232 22903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42507: connect: Connection refused (error 111)
I20260620 02:57:55.300796 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.300482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42507 (local address 127.0.0.1:42232)
0620 02:57:55.300617 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:55.300701 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 02:57:55.300973 22951 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42507: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42507: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260620 02:57:55.310480 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:42507
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36041,127.0.97.61:40233,127.0.97.60:42507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:57:55.350297 22951 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42507 } attempt: 1
I20260620 02:57:55.351646 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.351089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42507 (local address 127.0.0.1:42240)
0620 02:57:55.351323 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:55.351436 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 02:57:55.352780 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.352360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42507 (local address 127.0.0.1:42248)
0620 02:57:55.352546 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:55.352644 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 02:57:55.353060 22951 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42507: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42507: connect: Connection refused (error 111)
I20260620 02:57:55.421316 22951 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42507 } attempt: 2
I20260620 02:57:55.422628 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.422049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42507 (local address 127.0.0.1:42256)
0620 02:57:55.422274 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:55.422426 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 02:57:55.423672 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.423290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42507 (local address 127.0.0.1:42268)
0620 02:57:55.423448 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:55.423537 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 02:57:55.423882 22951 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42507: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42507: connect: Connection refused (error 111)
I20260620 02:57:55.445353 22877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } attempt: 5
Jun 20 02:57:55 dist-test-slave-8p8c krb5kdc[22797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 02:57:55.461236 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.446268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40233 (local address 127.0.0.1:40706)
0620 02:57:55.446480 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:55.446503 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:55.446759 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:55.447291 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:55.447295 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:55.447694 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:57:55.447878 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:55.447886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.449335 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.449339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:55.450234 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:55.450246 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.451262 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.451267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:55.451436 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:55.452168 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:57:55.452187 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:57:55.454214 (+ 2027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:57:55.458627 (+ 4413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:57:55.458631 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:57:55.458634 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:57:55.459031 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:57:55.459473 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:57:55.459476 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:57:55.459479 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:57:55.459619 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:57:55.460298 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:57:55.460303 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:57:55.460539 (+ 236us) client_negotiation.cc:769] Sending connection context
0620 02:57:55.460732 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 02:57:55.460959 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 02:57:55.461402 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.446359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40706 (local address 127.0.97.61:40233)
0620 02:57:55.446575 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:55.446581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:55.446605 (+ 24us) server_negotiation.cc:407] Connection header received
0620 02:57:55.446940 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:55.446946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:55.447048 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:55.447178 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:55.448042 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.449178 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.450427 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.451093 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.451224 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:55.454455 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:55.454462 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:55.454467 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:55.454509 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:55.458451 (+ 3942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:55.459198 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:55.459204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:55.459207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:55.459354 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:55.459729 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:55.459734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:55.459737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:55.460134 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:55.460281 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:55.460865 (+ 584us) server_negotiation.cc:299] Negotiation successful
0620 02:57:55.461095 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 02:57:55.462498 22877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42507 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:57:55.463652 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.463217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42507 (local address 127.0.0.1:42278)
0620 02:57:55.463399 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:55.463513 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 02:57:55.464697 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.464269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42507 (local address 127.0.0.1:42292)
0620 02:57:55.464409 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:55.464586 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 02:57:55.464957 22877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42507: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42507: connect: Connection refused (error 111)
I20260620 02:57:55.515134 22951 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42507 } attempt: 3
I20260620 02:57:55.516227 22877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42507 } attempt: 1
I20260620 02:57:55.516444 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.515866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42507 (local address 127.0.0.1:42298)
0620 02:57:55.516090 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:55.516246 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 02:57:55.517335 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.516852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42507 (local address 127.0.0.1:42310)
0620 02:57:55.517053 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:55.517151 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 02:57:55.517613 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.517170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42507 (local address 127.0.0.1:42322)
0620 02:57:55.517397 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:55.517487 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260620 02:57:55.517901 22951 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42507: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42507: connect: Connection refused (error 111)
I20260620 02:57:55.518673 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.518235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42507 (local address 127.0.0.1:42326)
0620 02:57:55.518437 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:55.518536 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 02:57:55.518941 22877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42507: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42507: connect: Connection refused (error 111)
I20260620 02:57:55.587168 22877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42507 } attempt: 2
I20260620 02:57:55.588449 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.587849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42507 (local address 127.0.0.1:42342)
0620 02:57:55.588112 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:55.588235 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 02:57:55.589725 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.589279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42507 (local address 127.0.0.1:42352)
0620 02:57:55.589448 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:55.589554 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 02:57:55.589982 22877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42507: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42507: connect: Connection refused (error 111)
W20260620 02:57:55.650926 22958 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 02:57:55.651304 22958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:55.651405 22958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:55.651563 22958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:55.659061 22958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:57:55.659163 22958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:55.659261 22958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:55.659361 22958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:57:55.659469 22958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:57:55.661155 22951 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42507 } attempt: 4
I20260620 02:57:55.662320 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.661835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42507 (local address 127.0.0.1:42362)
0620 02:57:55.662069 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:55.662164 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 02:57:55.663363 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.663047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42507 (local address 127.0.0.1:42366)
0620 02:57:55.663190 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:55.663260 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 02:57:55.663654 22951 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42507: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42507: connect: Connection refused (error 111)
I20260620 02:57:55.670316 22958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36041,127.0.97.61:40233,127.0.97.60:42507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:42507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:55.671702 22958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:55.673010 22958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:55.686165 22966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:55.686242 22964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:55.686164 22963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:55.687237 22958 server_base.cc:1034] running on GCE node
I20260620 02:57:55.688139 22958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:55.689653 22958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:55.690934 22958 hybrid_clock.cc:648] HybridClock initialized: now 1781924275690860 us; error 54 us; skew 500 ppm
I20260620 02:57:55.691355 22958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:57:55 dist-test-slave-8p8c krb5kdc[22797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:57:55.694868 22958 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260620 02:57:55.696391 22958 webserver.cc:460] Webserver started at http://127.0.97.60:38863/ using document root <none> and password file <none>
I20260620 02:57:55.696282 22877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42507 } attempt: 3
I20260620 02:57:55.697258 22958 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:55.697399 22958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:55.697441 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.696870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42507 (local address 127.0.0.1:42368)
0620 02:57:55.697131 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:55.697241 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 02:57:55.697813 22958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:55.698741 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.698140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42507 (local address 127.0.0.1:42384)
0620 02:57:55.698360 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:55.698568 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260620 02:57:55.699064 22877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42507: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42507: connect: Connection refused (error 111)
I20260620 02:57:55.701721 22958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-2/data/instance:
uuid: "ee82e2c51f784062afb3709c2ea3364a"
format_stamp: "Formatted at 2026-06-20 02:57:55 on dist-test-slave-8p8c"
server_key: "984ac108d322f79b62e122cc7c88c158"
server_key_iv: "f5cce6b7836349e59c538bef00534542"
server_key_version: "encryptionkey@0"
I20260620 02:57:55.702667 22958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-2/wal/instance:
uuid: "ee82e2c51f784062afb3709c2ea3364a"
format_stamp: "Formatted at 2026-06-20 02:57:55 on dist-test-slave-8p8c"
server_key: "984ac108d322f79b62e122cc7c88c158"
server_key_iv: "f5cce6b7836349e59c538bef00534542"
server_key_version: "encryptionkey@0"
I20260620 02:57:55.708563 22958 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 02:57:55.712630 22973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:55.714481 22958 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260620 02:57:55.714710 22958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-2/wal
uuid: "ee82e2c51f784062afb3709c2ea3364a"
format_stamp: "Formatted at 2026-06-20 02:57:55 on dist-test-slave-8p8c"
server_key: "984ac108d322f79b62e122cc7c88c158"
server_key_iv: "f5cce6b7836349e59c538bef00534542"
server_key_version: "encryptionkey@0"
I20260620 02:57:55.714985 22958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:55.737552 22958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:55.742170 22958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:55.742499 22958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:55.762234 22958 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:42507
I20260620 02:57:55.762264 23024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:42507 every 8 connection(s)
I20260620 02:57:55.763585 22958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-2/data/info.pb
I20260620 02:57:55.765973 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 22958
I20260620 02:57:55.766173 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781923914418765-388-0/minicluster-data/master-2/wal/instance
I20260620 02:57:55.766723 388 external_mini_cluster.cc:1428] Setting key b260eb22f908ddb148cb08e656a2eb72
I20260620 02:57:55.767946 23025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:55.777835 23025 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36041 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:57:55 dist-test-slave-8p8c krb5kdc[22797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 02:57:55 dist-test-slave-8p8c krb5kdc[22797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:57:55.790123 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.768619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42392 (local address 127.0.97.60:42507)
0620 02:57:55.769372 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:55.769383 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:55.769422 (+ 39us) server_negotiation.cc:407] Connection header received
0620 02:57:55.771010 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:55.771038 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:55.771413 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:55.771742 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:55.773532 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.774954 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.776890 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.777722 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.777857 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:55.783081 (+ 5224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:55.783089 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:55.783100 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:55.783138 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:55.786073 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:55.786849 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:55.786856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:55.786862 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:55.787020 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:55.787401 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:55.787407 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:55.787410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:55.787918 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:55.788292 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:55.788680 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 02:57:55.789080 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":355,"threads_started":1}
I20260620 02:57:55.796453 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.781211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36041 (local address 127.0.0.1:49992)
0620 02:57:55.782224 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:55.782261 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:55.782475 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:55.782879 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:55.782885 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:55.783569 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:57:55.783831 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:55.783839 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.784921 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.784926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:55.786049 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:55.786065 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.786869 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.786875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:55.787049 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:55.787639 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:57:55.787661 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:57:55.789808 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:57:55.793539 (+ 3731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:57:55.793550 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:57:55.793557 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:57:55.794203 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:57:55.794646 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:57:55.794653 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:57:55.794657 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:57:55.794797 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:57:55.795346 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:57:55.795354 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:57:55.795516 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 02:57:55.795699 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 02:57:55.796011 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":677,"threads_started":1}
I20260620 02:57:55.796471 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.781151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49992 (local address 127.0.97.62:36041)
0620 02:57:55.781708 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:55.781712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:55.782395 (+ 683us) server_negotiation.cc:407] Connection header received
0620 02:57:55.782625 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:55.782629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:55.782696 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:55.782815 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:55.783984 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.784761 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.786195 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.786659 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.786819 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:55.789998 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:55.790003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:55.790006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:55.790046 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:55.793358 (+ 3312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:55.794331 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:55.794336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:55.794340 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:55.794532 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:55.794911 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:55.794917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:55.794920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:55.795192 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:55.795350 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:55.795806 (+ 456us) server_negotiation.cc:299] Negotiation successful
0620 02:57:55.796011 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":330,"threads_started":1}
I20260620 02:57:55.798069 23025 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:57:55 dist-test-slave-8p8c krb5kdc[22797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Jun 20 02:57:55 dist-test-slave-8p8c krb5kdc[22797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 02:57:55.810472 23031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.793561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50006 (local address 127.0.97.62:36041)
0620 02:57:55.794407 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:55.794420 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:55.794446 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:57:55.794495 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:55.794499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:55.794557 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:55.794691 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:55.796104 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.797459 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.799223 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.799887 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.800037 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:55.804606 (+ 4569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:55.804611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:55.804615 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:55.804643 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:55.807526 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:55.808405 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:55.808409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:55.808412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:55.808520 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:55.808934 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:55.808938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:55.808940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:55.809208 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:55.809442 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:55.809787 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 02:57:55.810027 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":478,"threads_started":1}
I20260620 02:57:55.812268 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.798919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40233 (local address 127.0.0.1:40718)
0620 02:57:55.799182 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:55.799203 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:55.799285 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:55.799674 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:55.799678 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:55.799991 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:57:55.800132 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:55.800140 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.801106 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.801113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:55.801986 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:55.801996 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.802766 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.802771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:55.802966 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:55.803555 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:57:55.803580 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:57:55.805963 (+ 2383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:57:55.809517 (+ 3554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:57:55.809522 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:57:55.809526 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:57:55.810012 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:57:55.810539 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:57:55.810545 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:57:55.810548 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:57:55.810726 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:57:55.811311 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:57:55.811316 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:57:55.811493 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 02:57:55.811653 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 02:57:55.811917 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 02:57:55.812428 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.799125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40718 (local address 127.0.97.61:40233)
0620 02:57:55.799335 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:55.799342 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:55.799359 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:57:55.799415 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:55.799421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:55.799505 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:55.799630 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:55.800293 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.800961 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.802136 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.802619 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.802754 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:55.806165 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:55.806170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:55.806173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:55.806207 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:55.809392 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:55.810196 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:55.810202 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:55.810206 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:55.810417 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:55.810851 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:55.810855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:55.810857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:55.811149 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:55.811295 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:55.811750 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 02:57:55.811982 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 02:57:55.813373 23025 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42507 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:57:55 dist-test-slave-8p8c krb5kdc[22797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 02:57:55 dist-test-slave-8p8c krb5kdc[22797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 02:57:55.828730 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.814311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42408 (local address 127.0.97.60:42507)
0620 02:57:55.814556 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:55.814566 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:55.814592 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:57:55.814723 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:55.814732 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:55.814825 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:55.814953 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:55.815779 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.817289 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.818798 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.819192 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.819325 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:55.822146 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:55.822152 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:55.822156 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:55.822193 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:55.825840 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:55.826615 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:55.826622 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:55.826627 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:55.826777 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:55.827137 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:55.827143 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:55.827146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:55.827493 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:55.827638 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:55.828084 (+ 446us) server_negotiation.cc:299] Negotiation successful
0620 02:57:55.828260 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 02:57:55.829005 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.814032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42507 (local address 127.0.0.1:42408)
0620 02:57:55.814485 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:55.814501 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:55.814607 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:55.814981 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:55.814985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:55.815456 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:57:55.815629 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:55.815640 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.817433 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.817439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:55.818663 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:55.818680 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.819357 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.819363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:55.819586 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:55.820131 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:57:55.820146 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:57:55.821978 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:57:55.826009 (+ 4031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:57:55.826020 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:57:55.826023 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:57:55.826492 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:57:55.826891 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:57:55.826894 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:57:55.826897 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:57:55.827021 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:57:55.827655 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:57:55.827662 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:57:55.827841 (+ 179us) client_negotiation.cc:769] Sending connection context
0620 02:57:55.828002 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 02:57:55.828523 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":123}
I20260620 02:57:55.830577 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.812444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40732 (local address 127.0.97.61:40233)
0620 02:57:55.812766 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:55.812772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:55.812789 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:57:55.812841 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:55.812847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:55.812937 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:55.813069 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:55.814786 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.815856 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.817024 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.817521 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.817640 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:55.824132 (+ 6492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:55.824140 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:55.824145 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:55.824179 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:55.827427 (+ 3248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:55.828356 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:55.828360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:55.828363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:55.828514 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:55.829011 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:55.829019 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:55.829026 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:55.829347 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:55.829681 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:55.829952 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 02:57:55.830202 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":12319,"threads_started":1}
I20260620 02:57:55.835023 23025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee82e2c51f784062afb3709c2ea3364a: Bootstrap starting.
I20260620 02:57:55.835821 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:57:55.838330 23025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee82e2c51f784062afb3709c2ea3364a: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:55.839370 23025 log.cc:824] T 00000000000000000000000000000000 P ee82e2c51f784062afb3709c2ea3364a: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:55.838691 22852 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5e24609ff86d44a6aeea289d139fbf53"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50006
I20260620 02:57:55.842005 23025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee82e2c51f784062afb3709c2ea3364a: No bootstrap required, opened a new log
I20260620 02:57:55.849095 23025 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee82e2c51f784062afb3709c2ea3364a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e24609ff86d44a6aeea289d139fbf53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36041 } } peers { permanent_uuid: "eb8a95ae2e9346319372e0078599ec5a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "ee82e2c51f784062afb3709c2ea3364a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42507 } }
I20260620 02:57:55.849469 23025 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee82e2c51f784062afb3709c2ea3364a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee82e2c51f784062afb3709c2ea3364a, State: Initialized, Role: FOLLOWER
I20260620 02:57:55.849932 23025 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee82e2c51f784062afb3709c2ea3364a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e24609ff86d44a6aeea289d139fbf53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36041 } } peers { permanent_uuid: "eb8a95ae2e9346319372e0078599ec5a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "ee82e2c51f784062afb3709c2ea3364a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42507 } }
I20260620 02:57:55.850904 23034 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee82e2c51f784062afb3709c2ea3364a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e24609ff86d44a6aeea289d139fbf53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36041 } } peers { permanent_uuid: "eb8a95ae2e9346319372e0078599ec5a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "ee82e2c51f784062afb3709c2ea3364a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42507 } } }
I20260620 02:57:55.851303 23034 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee82e2c51f784062afb3709c2ea3364a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:57:55.852046 23025 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee82e2c51f784062afb3709c2ea3364a [sys.catalog]: configured and running, proceeding with master startup.
W20260620 02:57:55.861891 23045 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee82e2c51f784062afb3709c2ea3364a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:57:55.862037 23045 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:57:55.859701 22958 master_runner.cc:186] Error getting master registration for 127.0.97.62:36041: OK, instance_id { permanent_uuid: "5e24609ff86d44a6aeea289d139fbf53" instance_seqno: 1781924274733093 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:57:55.871332 22958 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:57:55.876302 22877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42507 } attempt: 4
Jun 20 02:57:55 dist-test-slave-8p8c krb5kdc[22797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 02:57:55.892238 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.877076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42507 (local address 127.0.0.1:42412)
0620 02:57:55.877379 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:55.877401 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:55.877550 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:55.877908 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:55.877913 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:55.878338 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:57:55.878580 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:55.878589 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.879941 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.879946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:55.880947 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:55.880959 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.881787 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.881793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:55.882007 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:55.882736 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:57:55.882765 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:57:55.885452 (+ 2687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:57:55.889326 (+ 3874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:57:55.889331 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:57:55.889336 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:57:55.889843 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:57:55.890229 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:57:55.890234 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:57:55.890238 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:57:55.890429 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:57:55.891138 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:57:55.891143 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:57:55.891351 (+ 208us) client_negotiation.cc:769] Sending connection context
0620 02:57:55.891506 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 02:57:55.891825 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 02:57:55.892249 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.877368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42412 (local address 127.0.97.60:42507)
0620 02:57:55.877603 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:55.877610 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:55.877632 (+ 22us) server_negotiation.cc:407] Connection header received
0620 02:57:55.877695 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:55.877700 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:55.877772 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:55.877874 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:55.878723 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.879772 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.881152 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.881618 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.881750 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:55.885645 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:55.885653 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:55.885657 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:55.885687 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:55.889180 (+ 3493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:55.889963 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:55.889969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:55.889972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:55.890110 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:55.890551 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:55.890555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:55.890559 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:55.891002 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:55.891154 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:55.891662 (+ 508us) server_negotiation.cc:299] Negotiation successful
0620 02:57:55.891882 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 02:57:55.898136 22877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e24609ff86d44a6aeea289d139fbf53: Bootstrap starting.
I20260620 02:57:55.902428 22877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e24609ff86d44a6aeea289d139fbf53: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:55.903541 22877 log.cc:824] T 00000000000000000000000000000000 P 5e24609ff86d44a6aeea289d139fbf53: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:55.906461 22877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e24609ff86d44a6aeea289d139fbf53: No bootstrap required, opened a new log
I20260620 02:57:55.912647 22877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e24609ff86d44a6aeea289d139fbf53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e24609ff86d44a6aeea289d139fbf53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36041 } } peers { permanent_uuid: "eb8a95ae2e9346319372e0078599ec5a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "ee82e2c51f784062afb3709c2ea3364a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42507 } }
I20260620 02:57:55.913117 22877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e24609ff86d44a6aeea289d139fbf53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e24609ff86d44a6aeea289d139fbf53, State: Initialized, Role: FOLLOWER
I20260620 02:57:55.913605 22877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e24609ff86d44a6aeea289d139fbf53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e24609ff86d44a6aeea289d139fbf53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36041 } } peers { permanent_uuid: "eb8a95ae2e9346319372e0078599ec5a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "ee82e2c51f784062afb3709c2ea3364a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42507 } }
I20260620 02:57:55.914645 23048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e24609ff86d44a6aeea289d139fbf53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e24609ff86d44a6aeea289d139fbf53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36041 } } peers { permanent_uuid: "eb8a95ae2e9346319372e0078599ec5a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "ee82e2c51f784062afb3709c2ea3364a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42507 } } }
I20260620 02:57:55.915153 23048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e24609ff86d44a6aeea289d139fbf53 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:57:55.915903 22877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e24609ff86d44a6aeea289d139fbf53 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:57:55.916039 22852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e24609ff86d44a6aeea289d139fbf53 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 02:57:55.916255 22852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e24609ff86d44a6aeea289d139fbf53 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:55.920198 22852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e24609ff86d44a6aeea289d139fbf53 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e24609ff86d44a6aeea289d139fbf53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36041 } } peers { permanent_uuid: "eb8a95ae2e9346319372e0078599ec5a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "ee82e2c51f784062afb3709c2ea3364a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42507 } }
I20260620 02:57:55.921950 22852 leader_election.cc:288] T 00000000000000000000000000000000 P 5e24609ff86d44a6aeea289d139fbf53 [CANDIDATE]: Term 1 election: Requested vote from peers eb8a95ae2e9346319372e0078599ec5a (127.0.97.61:40233), ee82e2c51f784062afb3709c2ea3364a (127.0.97.60:42507)
I20260620 02:57:55.924502 23000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e24609ff86d44a6aeea289d139fbf53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ee82e2c51f784062afb3709c2ea3364a"
I20260620 02:57:55.925041 23000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee82e2c51f784062afb3709c2ea3364a [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:55.928810 23000 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ee82e2c51f784062afb3709c2ea3364a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e24609ff86d44a6aeea289d139fbf53 in term 1.
I20260620 02:57:55.929605 22829 leader_election.cc:302] T 00000000000000000000000000000000 P 5e24609ff86d44a6aeea289d139fbf53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e24609ff86d44a6aeea289d139fbf53, ee82e2c51f784062afb3709c2ea3364a; no voters:
I20260620 02:57:55.930400 23048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e24609ff86d44a6aeea289d139fbf53 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:57:55.942831 23048 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e24609ff86d44a6aeea289d139fbf53 [term 1 LEADER]: Becoming Leader. State: Replica: 5e24609ff86d44a6aeea289d139fbf53, State: Running, Role: LEADER
I20260620 02:57:55.944788 23048 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e24609ff86d44a6aeea289d139fbf53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e24609ff86d44a6aeea289d139fbf53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36041 } } peers { permanent_uuid: "eb8a95ae2e9346319372e0078599ec5a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "ee82e2c51f784062afb3709c2ea3364a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42507 } }
I20260620 02:57:55.962885 22951 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42507 } attempt: 5
W20260620 02:57:55.972581 22831 tablet.cc:2369] T 00000000000000000000000000000000 P 5e24609ff86d44a6aeea289d139fbf53: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 02:57:55 dist-test-slave-8p8c krb5kdc[22797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 02:57:55.992658 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.963665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42507 (local address 127.0.0.1:42414)
0620 02:57:55.964019 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:55.964045 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:55.964180 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:55.964702 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:55.964712 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:55.965558 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:57:55.965997 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:55.966019 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.968020 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.968027 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:55.969901 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:55.969923 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.971780 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.971817 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:55.972255 (+ 438us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:55.973350 (+ 1095us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:57:55.973407 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:57:55.978630 (+ 5223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:57:55.983592 (+ 4962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:57:55.983599 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:57:55.983605 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:57:55.984486 (+ 881us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:57:55.985277 (+ 791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:57:55.985283 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:57:55.985287 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:57:55.985650 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:57:55.987273 (+ 1623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:57:55.987282 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:57:55.987687 (+ 405us) client_negotiation.cc:769] Sending connection context
0620 02:57:55.987938 (+ 251us) client_negotiation.cc:240] Negotiation successful
0620 02:57:55.991350 (+ 3412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200}
I20260620 02:57:55.993307 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.963877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42414 (local address 127.0.97.60:42507)
0620 02:57:55.964118 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:55.964127 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:55.964199 (+ 72us) server_negotiation.cc:407] Connection header received
0620 02:57:55.964386 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:55.964395 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:55.964501 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:55.964631 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:55.966278 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.967757 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.970169 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.971158 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.971391 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:55.978869 (+ 7478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:55.978878 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:55.978884 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:55.978929 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:55.983351 (+ 4422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:55.984689 (+ 1338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:55.984696 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:55.984700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:55.985040 (+ 340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:55.985944 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:55.985982 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:55.985990 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:55.986982 (+ 992us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:55.988381 (+ 1399us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:55.989043 (+ 662us) server_negotiation.cc:299] Negotiation successful
0620 02:57:55.992004 (+ 2961us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260620 02:57:55.994812 22979 tablet.cc:2369] T 00000000000000000000000000000000 P ee82e2c51f784062afb3709c2ea3364a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:57:56.000092 22951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb8a95ae2e9346319372e0078599ec5a: Bootstrap starting.
I20260620 02:57:56.006636 22951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb8a95ae2e9346319372e0078599ec5a: Neither blocks nor log segments found. Creating new log.
I20260620 02:57:56.010874 23056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e24609ff86d44a6aeea289d139fbf53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e24609ff86d44a6aeea289d139fbf53. Latest consensus state: current_term: 1 leader_uuid: "5e24609ff86d44a6aeea289d139fbf53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e24609ff86d44a6aeea289d139fbf53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36041 } } peers { permanent_uuid: "eb8a95ae2e9346319372e0078599ec5a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "ee82e2c51f784062afb3709c2ea3364a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42507 } } }
I20260620 02:57:56.011373 23056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e24609ff86d44a6aeea289d139fbf53 [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:56.018973 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.960613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40744 (local address 127.0.97.61:40233)
0620 02:57:55.960821 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:55.960828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:55.968159 (+ 7331us) server_negotiation.cc:407] Connection header received
0620 02:57:55.969311 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:55.969317 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:55.969399 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:55.969470 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:55.975458 (+ 5988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.976867 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.982604 (+ 5737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:55.983291 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:55.983395 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:55.986834 (+ 3439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:55.986841 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:55.986847 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:55.986885 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:55.991132 (+ 4247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:56.013842 (+ 22710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:56.013849 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:56.013852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:56.014069 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:56.016556 (+ 2487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:56.016564 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:56.016568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:56.017048 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:56.017390 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:56.018159 (+ 769us) server_negotiation.cc:299] Negotiation successful
0620 02:57:56.018503 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 02:57:56.020946 23069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:57:56.024981 22951 log.cc:824] T 00000000000000000000000000000000 P eb8a95ae2e9346319372e0078599ec5a: Log is configured to *not* fsync() on all Append() calls
I20260620 02:57:56.031487 22951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb8a95ae2e9346319372e0078599ec5a: No bootstrap required, opened a new log
I20260620 02:57:56.036465 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.968696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42422 (local address 127.0.97.60:42507)
0620 02:57:55.981513 (+ 12817us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:55.981519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:55.990982 (+ 9463us) server_negotiation.cc:407] Connection header received
0620 02:57:55.991362 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:55.991368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:55.991792 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:55.991933 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:55.994061 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:56.003809 (+ 9748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:56.007521 (+ 3712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:56.008342 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:56.008490 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:56.015257 (+ 6767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:56.015299 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:56.015306 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:56.015390 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:56.031108 (+ 15718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:56.033391 (+ 2283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:56.033398 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:56.033402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:56.033581 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:56.034045 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:56.034050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:56.034054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:56.034481 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:56.035663 (+ 1182us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:56.035790 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 02:57:56.036019 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12654,"thread_start_us":12813,"threads_started":1}
I20260620 02:57:56.044896 22951 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb8a95ae2e9346319372e0078599ec5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e24609ff86d44a6aeea289d139fbf53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36041 } } peers { permanent_uuid: "eb8a95ae2e9346319372e0078599ec5a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "ee82e2c51f784062afb3709c2ea3364a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42507 } }
I20260620 02:57:56.045447 22951 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb8a95ae2e9346319372e0078599ec5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb8a95ae2e9346319372e0078599ec5a, State: Initialized, Role: FOLLOWER
I20260620 02:57:56.046056 22951 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb8a95ae2e9346319372e0078599ec5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e24609ff86d44a6aeea289d139fbf53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36041 } } peers { permanent_uuid: "eb8a95ae2e9346319372e0078599ec5a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "ee82e2c51f784062afb3709c2ea3364a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42507 } }
I20260620 02:57:56.030201 22810 master_runner.cc:186] Error getting master registration for 127.0.97.61:40233: OK, instance_id { permanent_uuid: "eb8a95ae2e9346319372e0078599ec5a" instance_seqno: 1781924275235075 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:57:56.047199 22810 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:57:56.053030 23031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:55.960663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50014 (local address 127.0.97.62:36041)
0620 02:57:55.960889 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:55.960897 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:56.012595 (+ 51698us) server_negotiation.cc:407] Connection header received
0620 02:57:56.012963 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:56.012972 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:56.013075 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:56.013168 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:56.019845 (+ 6677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:56.021109 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:56.027239 (+ 6130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:56.028005 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:56.028159 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:56.032057 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:56.032065 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:56.032070 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:56.032112 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:56.036516 (+ 4404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:56.048509 (+ 11993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:56.048519 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:56.048530 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:56.048806 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:56.049414 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:56.049421 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:56.049427 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:56.049944 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:56.050313 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:56.051921 (+ 1608us) server_negotiation.cc:299] Negotiation successful
0620 02:57:56.052422 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 02:57:56.053522 23069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:57:56.055779 23073 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb8a95ae2e9346319372e0078599ec5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e24609ff86d44a6aeea289d139fbf53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36041 } } peers { permanent_uuid: "eb8a95ae2e9346319372e0078599ec5a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "ee82e2c51f784062afb3709c2ea3364a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42507 } } }
I20260620 02:57:56.056385 23073 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb8a95ae2e9346319372e0078599ec5a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:57:56.057231 22951 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb8a95ae2e9346319372e0078599ec5a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:57:56.057264 22926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e24609ff86d44a6aeea289d139fbf53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "eb8a95ae2e9346319372e0078599ec5a"
I20260620 02:57:56.057781 22926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb8a95ae2e9346319372e0078599ec5a [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:57:56.061566 22926 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eb8a95ae2e9346319372e0078599ec5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e24609ff86d44a6aeea289d139fbf53 in term 1.
I20260620 02:57:56.075091 23000 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ee82e2c51f784062afb3709c2ea3364a [term 1 FOLLOWER]: Refusing update from remote peer 5e24609ff86d44a6aeea289d139fbf53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:57:56.076282 22926 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eb8a95ae2e9346319372e0078599ec5a [term 1 FOLLOWER]: Refusing update from remote peer 5e24609ff86d44a6aeea289d139fbf53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:57:56.080256 23056 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5e24609ff86d44a6aeea289d139fbf53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb8a95ae2e9346319372e0078599ec5a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:57:56.080600 23048 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5e24609ff86d44a6aeea289d139fbf53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee82e2c51f784062afb3709c2ea3364a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:57:56.102099 23034 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee82e2c51f784062afb3709c2ea3364a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e24609ff86d44a6aeea289d139fbf53. Latest consensus state: current_term: 1 leader_uuid: "5e24609ff86d44a6aeea289d139fbf53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e24609ff86d44a6aeea289d139fbf53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36041 } } peers { permanent_uuid: "eb8a95ae2e9346319372e0078599ec5a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "ee82e2c51f784062afb3709c2ea3364a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42507 } } }
I20260620 02:57:56.102452 23034 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee82e2c51f784062afb3709c2ea3364a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:57:56.107086 23073 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb8a95ae2e9346319372e0078599ec5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e24609ff86d44a6aeea289d139fbf53. Latest consensus state: current_term: 1 leader_uuid: "5e24609ff86d44a6aeea289d139fbf53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e24609ff86d44a6aeea289d139fbf53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36041 } } peers { permanent_uuid: "eb8a95ae2e9346319372e0078599ec5a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "ee82e2c51f784062afb3709c2ea3364a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42507 } } }
I20260620 02:57:56.107517 23073 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb8a95ae2e9346319372e0078599ec5a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:57:56.107867 23084 mvcc.cc:205] Tried to move back new op lower bound from 7298761834774847488 to 7298761834290274304. Current Snapshot: MvccSnapshot[applied={T|T < 7298761834774847488}]
I20260620 02:57:56.116513 23034 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee82e2c51f784062afb3709c2ea3364a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e24609ff86d44a6aeea289d139fbf53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e24609ff86d44a6aeea289d139fbf53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36041 } } peers { permanent_uuid: "eb8a95ae2e9346319372e0078599ec5a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "ee82e2c51f784062afb3709c2ea3364a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42507 } } }
I20260620 02:57:56.116819 23034 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee82e2c51f784062afb3709c2ea3364a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:57:56.122581 23081 mvcc.cc:205] Tried to move back new op lower bound from 7298761834774847488 to 7298761834290274304. Current Snapshot: MvccSnapshot[applied={T|T < 7298761834774847488}]
I20260620 02:57:56.126180 23086 mvcc.cc:205] Tried to move back new op lower bound from 7298761834774847488 to 7298761834290274304. Current Snapshot: MvccSnapshot[applied={T|T < 7298761834774847488}]
I20260620 02:57:56.130911 23073 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb8a95ae2e9346319372e0078599ec5a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e24609ff86d44a6aeea289d139fbf53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e24609ff86d44a6aeea289d139fbf53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36041 } } peers { permanent_uuid: "eb8a95ae2e9346319372e0078599ec5a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "ee82e2c51f784062afb3709c2ea3364a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42507 } } }
I20260620 02:57:56.131174 23073 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb8a95ae2e9346319372e0078599ec5a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:57:56.134747 23048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e24609ff86d44a6aeea289d139fbf53 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e24609ff86d44a6aeea289d139fbf53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e24609ff86d44a6aeea289d139fbf53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36041 } } peers { permanent_uuid: "eb8a95ae2e9346319372e0078599ec5a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "ee82e2c51f784062afb3709c2ea3364a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42507 } } }
I20260620 02:57:56.135111 23048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e24609ff86d44a6aeea289d139fbf53 [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:56.135723 23048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e24609ff86d44a6aeea289d139fbf53 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e24609ff86d44a6aeea289d139fbf53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e24609ff86d44a6aeea289d139fbf53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36041 } } peers { permanent_uuid: "eb8a95ae2e9346319372e0078599ec5a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40233 } } peers { permanent_uuid: "ee82e2c51f784062afb3709c2ea3364a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42507 } } }
I20260620 02:57:56.135994 23048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e24609ff86d44a6aeea289d139fbf53 [sys.catalog]: This master's current role is: LEADER
I20260620 02:57:56.135649 23091 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 02:57:56.136582 23069 catalog_manager.cc:1347] Generated new cluster ID: e2dcf3fb61c34535be1e2075ed7dad74
I20260620 02:57:56.136696 23069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:57:56.162815 23069 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:57:56.164796 23069 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:57:56.181120 23069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e24609ff86d44a6aeea289d139fbf53: Generated new TSK 0
I20260620 02:57:56.182649 23069 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 02:57:56.252012 23031 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:57:56.237186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50018 (local address 127.0.97.62:36041)
0620 02:57:56.237376 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:56.237383 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:56.249915 (+ 12532us) server_negotiation.cc:407] Connection header received
0620 02:57:56.250413 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:56.250420 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:56.250502 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:56.250574 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:56.251708 (+ 1134us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50018: BlockingRecv error: recv got EOF from 127.0.0.1:50018 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 02:57:56.254961 22955 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:57:56.236845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40754 (local address 127.0.97.61:40233)
0620 02:57:56.237057 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:56.237064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:56.253084 (+ 16020us) server_negotiation.cc:407] Connection header received
0620 02:57:56.253358 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:56.253365 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:56.253478 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:56.253591 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:56.254729 (+ 1138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40754: BlockingRecv error: recv got EOF from 127.0.0.1:40754 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 02:57:56.265918 23067 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:57:56.263275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42426 (local address 127.0.97.60:42507)
0620 02:57:56.263455 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:56.263460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:56.263930 (+ 470us) server_negotiation.cc:407] Connection header received
0620 02:57:56.264209 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:56.264215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:56.264322 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:56.264405 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:56.265501 (+ 1096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42426: BlockingRecv error: recv got EOF from 127.0.0.1:42426 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 02:57:56.287241 22955 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:57:56.284816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40768 (local address 127.0.97.61:40233)
0620 02:57:56.285044 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:56.285054 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:56.285579 (+ 525us) server_negotiation.cc:407] Connection header received
0620 02:57:56.285855 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:56.285860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:56.285965 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:56.286090 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:56.286945 (+ 855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40768: BlockingRecv error: recv got EOF from 127.0.0.1:40768 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 02:57:56.293231 22955 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:57:56.291233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40772 (local address 127.0.97.61:40233)
0620 02:57:56.291445 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:56.291453 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:56.291959 (+ 506us) server_negotiation.cc:407] Connection header received
0620 02:57:56.292162 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:56.292168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:56.292242 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:56.292345 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:56.293018 (+ 673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40772: BlockingRecv error: recv got EOF from 127.0.0.1:40772 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 02:57:56.300335 22955 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:57:56.298234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40786 (local address 127.0.97.61:40233)
0620 02:57:56.298421 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:56.298429 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:56.298943 (+ 514us) server_negotiation.cc:407] Connection header received
0620 02:57:56.299151 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:56.299157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:56.299243 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:56.299345 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:56.300087 (+ 742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40786: BlockingRecv error: recv got EOF from 127.0.0.1:40786 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 02:57:56.313716 22955 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:57:56.311475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40800 (local address 127.0.97.61:40233)
0620 02:57:56.311727 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:56.311736 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:56.312273 (+ 537us) server_negotiation.cc:407] Connection header received
0620 02:57:56.312494 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:56.312500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:56.312594 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:56.312704 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:56.313462 (+ 758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40800: BlockingRecv error: recv got EOF from 127.0.0.1:40800 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260620 02:57:56.327462 22955 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:57:56.325115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40804 (local address 127.0.97.61:40233)
0620 02:57:56.325367 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:56.325377 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:56.326052 (+ 675us) server_negotiation.cc:407] Connection header received
0620 02:57:56.326249 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:56.326255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:56.326339 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:56.326480 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:56.327231 (+ 751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40804: BlockingRecv error: recv got EOF from 127.0.0.1:40804 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260620 02:57:56.348893 22955 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:57:56.346614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40810 (local address 127.0.97.61:40233)
0620 02:57:56.346903 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:56.346913 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:56.347445 (+ 532us) server_negotiation.cc:407] Connection header received
0620 02:57:56.347684 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:56.347688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:56.347776 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:56.347886 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:56.348648 (+ 762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40810: BlockingRecv error: recv got EOF from 127.0.0.1:40810 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260620 02:57:56.386413 22955 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:57:56.384163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40816 (local address 127.0.97.61:40233)
0620 02:57:56.384404 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:56.384414 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:56.384973 (+ 559us) server_negotiation.cc:407] Connection header received
0620 02:57:56.385205 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:56.385210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:56.385286 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:56.385396 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:56.386142 (+ 746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40816: BlockingRecv error: recv got EOF from 127.0.0.1:40816 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 02:57:56.456188 22955 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:57:56.453723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40820 (local address 127.0.97.61:40233)
0620 02:57:56.453992 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:56.454000 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:56.454621 (+ 621us) server_negotiation.cc:407] Connection header received
0620 02:57:56.454898 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:56.454909 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:56.455002 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:56.455119 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:56.455948 (+ 829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40820: BlockingRecv error: recv got EOF from 127.0.0.1:40820 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 02:57:56.589520 22955 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:57:56.587596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40822 (local address 127.0.97.61:40233)
0620 02:57:56.587860 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:56.587866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:56.588033 (+ 167us) server_negotiation.cc:407] Connection header received
0620 02:57:56.588274 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:56.588279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:56.588363 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:56.588466 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:56.589265 (+ 799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40822: BlockingRecv error: recv got EOF from 127.0.0.1:40822 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
W20260620 02:57:56.854272 22955 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:57:56.852359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40830 (local address 127.0.97.61:40233)
0620 02:57:56.852562 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:56.852567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:56.852825 (+ 258us) server_negotiation.cc:407] Connection header received
0620 02:57:56.853074 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:56.853078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:56.853153 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:56.853266 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:56.854043 (+ 777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40830: BlockingRecv error: recv got EOF from 127.0.0.1:40830 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 02:57:56.864562 23045 catalog_manager.cc:1259] Loaded cluster ID: e2dcf3fb61c34535be1e2075ed7dad74
I20260620 02:57:56.864676 23045 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ee82e2c51f784062afb3709c2ea3364a: loading cluster ID for follower catalog manager: success
I20260620 02:57:56.867311 23045 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ee82e2c51f784062afb3709c2ea3364a: acquiring CA information for follower catalog manager: success
I20260620 02:57:56.869129 23045 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ee82e2c51f784062afb3709c2ea3364a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:57:57.140513 23091 catalog_manager.cc:1259] Loaded cluster ID: e2dcf3fb61c34535be1e2075ed7dad74
I20260620 02:57:57.140743 23091 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eb8a95ae2e9346319372e0078599ec5a: loading cluster ID for follower catalog manager: success
I20260620 02:57:57.143622 23091 catalog_manager.cc:1574] T 00000000000000000000000000000000 P eb8a95ae2e9346319372e0078599ec5a: acquiring CA information for follower catalog manager: success
I20260620 02:57:57.145565 23091 catalog_manager.cc:1602] T 00000000000000000000000000000000 P eb8a95ae2e9346319372e0078599ec5a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:57:57.388866 23158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:57.382113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40834 (local address 127.0.97.61:40233)
0620 02:57:57.382940 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:57.382945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:57.382961 (+ 16us) server_negotiation.cc:407] Connection header received
0620 02:57:57.383023 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:57.383027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:57.383178 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:57.383325 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:57:57.383813 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:57.384727 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:57.386122 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:57.386829 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:57.386986 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:57.387568 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:57:57.388182 (+ 614us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:57:57.388346 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:57.388411 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 02:57:57.388510 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":506,"threads_started":1}
I20260620 02:57:57.415356 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:57.408548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42436 (local address 127.0.97.60:42507)
0620 02:57:57.409349 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:57.409356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:57.409380 (+ 24us) server_negotiation.cc:407] Connection header received
0620 02:57:57.409577 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:57.409581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:57.409825 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:57.410020 (+ 195us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:57:57.410492 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:57.411422 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:57.412845 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:57.413320 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:57.413488 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:57.414017 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:57:57.414670 (+ 653us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:57:57.414845 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:57.414911 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 02:57:57.415004 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":533,"threads_started":1}
I20260620 02:57:57.433831 23158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:57.422791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40850 (local address 127.0.97.61:40233)
0620 02:57:57.422982 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:57.422988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:57.424453 (+ 1465us) server_negotiation.cc:407] Connection header received
0620 02:57:57.424713 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:57.424719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:57.424831 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:57.424914 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:57:57.425834 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:57.427587 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:57.430633 (+ 3046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:57.431632 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:57.431771 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:57.432618 (+ 847us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:57:57.432803 (+ 185us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:57:57.432884 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:57.433227 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 02:57:57.433350 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:57:57.441746 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:57.424570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42448 (local address 127.0.97.60:42507)
0620 02:57:57.424757 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:57.424763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:57.434967 (+ 10204us) server_negotiation.cc:407] Connection header received
0620 02:57:57.435306 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:57.435316 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:57.435450 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:57.435531 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:57:57.436149 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:57.437257 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:57.439251 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:57.439852 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:57.439942 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:57.440890 (+ 948us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:57:57.441030 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:57:57.441091 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:57.441341 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 02:57:57.441439 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:57:57.443456 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:57.423004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50030 (local address 127.0.97.62:36041)
0620 02:57:57.435969 (+ 12965us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:57.435975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:57.435995 (+ 20us) server_negotiation.cc:407] Connection header received
0620 02:57:57.436116 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:57.436121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:57.436357 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:57.437228 (+ 871us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:57:57.437396 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:57.438486 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:57.440679 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:57.441341 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:57.441974 (+ 633us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:57.442184 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:57:57.442708 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:57:57.442815 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:57.442950 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 02:57:57.443032 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12830,"thread_start_us":13507,"threads_started":1}
I20260620 02:57:57.448820 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 22810
I20260620 02:57:57.450270 22872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:57:57.565310 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 22810
W20260620 02:57:57.585757 22903 connection.cc:535] server connection from 127.0.0.1:40706 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 02:57:57.586474 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 22884
I20260620 02:57:57.587612 22946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:57:57.694926 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 22884
I20260620 02:57:57.715236 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 22958
I20260620 02:57:57.716427 23020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:57:57.824517 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 22958
2026-06-20T02:57:57Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4936 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 02:57:57 dist-test-slave-8p8c krb5kdc[23178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 02:57:57 dist-test-slave-8p8c krb5kdc[23178](info): set up 2 sockets
Jun 20 02:57:57 dist-test-slave-8p8c krb5kdc[23178](info): commencing operation
krb5kdc: starting...
W20260620 02:57:58.959074 388 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.008s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/krb5kdc/test-user.keytab.
Jun 20 02:57:59 dist-test-slave-8p8c krb5kdc[23178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T02:57:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T02:57:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 02:57:59.238328 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36377
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36377,127.0.97.61:33505,127.0.97.60:43131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 02:57:59.571460 23191 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 02:57:59.571905 23191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:57:59.572014 23191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:57:59.572178 23191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:57:59.579543 23191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:57:59.579638 23191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:57:59.579730 23191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:57:59.579821 23191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:57:59.579910 23191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:57:59.589937 23191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36377,127.0.97.61:33505,127.0.97.60:43131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:57:59.591307 23191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:57:59.592666 23191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:57:59.604869 23199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:59.604920 23196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:57:59.604960 23197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:57:59.605808 23191 server_base.cc:1034] running on GCE node
I20260620 02:57:59.606741 23191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:57:59.608502 23191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:57:59.609802 23191 hybrid_clock.cc:648] HybridClock initialized: now 1781924279609733 us; error 63 us; skew 500 ppm
I20260620 02:57:59.610229 23191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:57:59 dist-test-slave-8p8c krb5kdc[23178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:57:59.613772 23191 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 02:57:59.615463 23191 webserver.cc:460] Webserver started at http://127.0.97.62:35497/ using document root <none> and password file <none>
I20260620 02:57:59.616267 23191 fs_manager.cc:359] Metadata directory not provided
I20260620 02:57:59.616413 23191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:57:59.616756 23191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:57:59.620095 23191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-0/data/instance:
uuid: "72350ceb10504a6aace53b7216b41263"
format_stamp: "Formatted at 2026-06-20 02:57:59 on dist-test-slave-8p8c"
server_key: "15192ae35323cd0ac302b034724e3f9b"
server_key_iv: "f2649aa9d2e4b32aaa0548c9c472c379"
server_key_version: "encryptionkey@0"
I20260620 02:57:59.620991 23191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-0/wal/instance:
uuid: "72350ceb10504a6aace53b7216b41263"
format_stamp: "Formatted at 2026-06-20 02:57:59 on dist-test-slave-8p8c"
server_key: "15192ae35323cd0ac302b034724e3f9b"
server_key_iv: "f2649aa9d2e4b32aaa0548c9c472c379"
server_key_version: "encryptionkey@0"
I20260620 02:57:59.627188 23191 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.006s
I20260620 02:57:59.630801 23206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:57:59.632309 23191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 02:57:59.632540 23191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-0/wal
uuid: "72350ceb10504a6aace53b7216b41263"
format_stamp: "Formatted at 2026-06-20 02:57:59 on dist-test-slave-8p8c"
server_key: "15192ae35323cd0ac302b034724e3f9b"
server_key_iv: "f2649aa9d2e4b32aaa0548c9c472c379"
server_key_version: "encryptionkey@0"
I20260620 02:57:59.632795 23191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:57:59.646894 23191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:57:59.651265 23191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:57:59.651563 23191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:57:59.672389 23191 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36377
I20260620 02:57:59.672446 23257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36377 every 8 connection(s)
I20260620 02:57:59.673892 23191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-0/data/info.pb
I20260620 02:57:59.674822 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 23191
I20260620 02:57:59.675046 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-0/wal/instance
I20260620 02:57:59.675716 388 external_mini_cluster.cc:1428] Setting key 3f3300c97909e720e9289a1e586415b1
I20260620 02:57:59.678426 23258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:57:59.687119 23258 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36377 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Jun 20 02:57:59 dist-test-slave-8p8c krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:57:59.719928 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:59.691522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46822 (local address 127.0.97.62:36377)
0620 02:57:59.692522 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0620 02:57:59.692537 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 02:57:59.693183 (+ 646us) server_negotiation.cc:407] Connection header received
0620 02:57:59.696667 (+ 3484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:57:59.696675 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:57:59.696984 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:57:59.697178 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:57:59.699024 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:59.700312 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:59.702836 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:57:59.703478 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:57:59.703630 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:59.711556 (+ 7926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:57:59.711571 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:57:59.711591 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:57:59.711637 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:57:59.715321 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:59.716299 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:59.716306 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:59.716309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:59.716461 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:57:59.716942 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:57:59.716945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:57:59.716947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:57:59.717475 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:57:59.717677 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 02:57:59.718513 (+ 836us) server_negotiation.cc:299] Negotiation successful
0620 02:57:59.718802 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":718,"thread_start_us":460,"threads_started":1}
I20260620 02:57:59.719954 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:59.691505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36377 (local address 127.0.0.1:46822)
0620 02:57:59.692922 (+ 1417us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:59.692981 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 02:57:59.696146 (+ 3165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:57:59.697266 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:57:59.697277 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:57:59.698151 (+ 874us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:57:59.698712 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:59.698744 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:59.700493 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:59.700500 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:59.702672 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:57:59.702691 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:57:59.703713 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:57:59.703719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:57:59.703849 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:57:59.708867 (+ 5018us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:57:59.708899 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:57:59.711256 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:57:59.715475 (+ 4219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:57:59.715489 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:57:59.715506 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:57:59.716109 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:57:59.716591 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:57:59.716598 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:57:59.716601 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:57:59.716826 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:57:59.717660 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:57:59.717672 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:57:59.717964 (+ 292us) client_negotiation.cc:769] Sending connection context
0620 02:57:59.718256 (+ 292us) client_negotiation.cc:240] Negotiation successful
0620 02:57:59.719001 (+ 745us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1053,"mutex_wait_us":152,"spinlock_wait_cycles":33152,"thread_start_us":810,"threads_started":1}
I20260620 02:57:59.723227 23258 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33505 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:57:59.724370 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:59.723872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33505 (local address 127.0.0.1:59068)
0620 02:57:59.724080 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:59.724218 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 02:57:59.724640 23207 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33505: connect: Connection refused (error 111)
I20260620 02:57:59.726351 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:59.725887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33505 (local address 127.0.0.1:59082)
0620 02:57:59.726073 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:59.726173 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 02:57:59.726588 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33505: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33505: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260620 02:57:59.742677 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:33505
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36377,127.0.97.61:33505,127.0.97.60:43131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:57:59.775979 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33505 } attempt: 1
I20260620 02:57:59.777256 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:59.776749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33505 (local address 127.0.0.1:59096)
0620 02:57:59.776970 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:59.777096 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 02:57:59.778273 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:59.777895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33505 (local address 127.0.0.1:59112)
0620 02:57:59.778043 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:59.778123 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 02:57:59.778514 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33505: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33505: connect: Connection refused (error 111)
I20260620 02:57:59.846849 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33505 } attempt: 2
I20260620 02:57:59.848057 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:59.847558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33505 (local address 127.0.0.1:59116)
0620 02:57:59.847824 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:59.847919 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 02:57:59.849414 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:59.848846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33505 (local address 127.0.0.1:59132)
0620 02:57:59.849204 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:59.849280 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244}
W20260620 02:57:59.849691 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33505: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33505: connect: Connection refused (error 111)
I20260620 02:57:59.940996 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33505 } attempt: 3
I20260620 02:57:59.942209 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:59.941723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33505 (local address 127.0.0.1:59140)
0620 02:57:59.941937 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:59.942042 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 02:57:59.943493 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:57:59.942996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33505 (local address 127.0.0.1:59152)
0620 02:57:59.943197 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 02:57:59.943303 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260620 02:57:59.943742 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33505: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33505: connect: Connection refused (error 111)
W20260620 02:58:00.077418 23265 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 02:58:00.077783 23265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:58:00.077906 23265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:58:00.078043 23265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:58:00.085275 23265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:58:00.085364 23265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:58:00.085469 23265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:58:00.085556 23265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:58:00.085672 23265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:58:00.086975 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33505 } attempt: 4
I20260620 02:58:00.088213 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.087626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33505 (local address 127.0.0.1:59168)
0620 02:58:00.087942 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:00.088051 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260620 02:58:00.089274 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.088874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33505 (local address 127.0.0.1:59180)
0620 02:58:00.089032 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:00.089125 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 02:58:00.089457 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33505: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33505: connect: Connection refused (error 111)
I20260620 02:58:00.096233 23265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36377,127.0.97.61:33505,127.0.97.60:43131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:33505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:58:00.097499 23265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:58:00.098857 23265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:58:00.110253 23270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:00.110385 23271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:00.110417 23273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:58:00.111341 23265 server_base.cc:1034] running on GCE node
I20260620 02:58:00.112039 23265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:58:00.113389 23265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:58:00.114637 23265 hybrid_clock.cc:648] HybridClock initialized: now 1781924280114575 us; error 33 us; skew 500 ppm
I20260620 02:58:00.114986 23265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 02:58:00 dist-test-slave-8p8c krb5kdc[23178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:58:00.118022 23265 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260620 02:58:00.119560 23265 webserver.cc:460] Webserver started at http://127.0.97.61:32791/ using document root <none> and password file <none>
I20260620 02:58:00.120249 23265 fs_manager.cc:359] Metadata directory not provided
I20260620 02:58:00.120375 23265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:58:00.120688 23265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:58:00.123672 23265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-1/data/instance:
uuid: "85cec0bb4a6844e0bd211af6fd0e2e61"
format_stamp: "Formatted at 2026-06-20 02:58:00 on dist-test-slave-8p8c"
server_key: "f4b891c4d517a8aa880d639c641e96bf"
server_key_iv: "444b67d511c3f64fa757f4c4a6cbfc0f"
server_key_version: "encryptionkey@0"
I20260620 02:58:00.124428 23265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-1/wal/instance:
uuid: "85cec0bb4a6844e0bd211af6fd0e2e61"
format_stamp: "Formatted at 2026-06-20 02:58:00 on dist-test-slave-8p8c"
server_key: "f4b891c4d517a8aa880d639c641e96bf"
server_key_iv: "444b67d511c3f64fa757f4c4a6cbfc0f"
server_key_version: "encryptionkey@0"
I20260620 02:58:00.129465 23265 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 02:58:00.132889 23280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:00.134234 23265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 02:58:00.134465 23265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-1/wal
uuid: "85cec0bb4a6844e0bd211af6fd0e2e61"
format_stamp: "Formatted at 2026-06-20 02:58:00 on dist-test-slave-8p8c"
server_key: "f4b891c4d517a8aa880d639c641e96bf"
server_key_iv: "444b67d511c3f64fa757f4c4a6cbfc0f"
server_key_version: "encryptionkey@0"
I20260620 02:58:00.134706 23265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:58:00.170154 23265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:58:00.174988 23265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:58:00.175266 23265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:58:00.195075 23265 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:33505
I20260620 02:58:00.195098 23331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:33505 every 8 connection(s)
I20260620 02:58:00.196483 23265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-1/data/info.pb
I20260620 02:58:00.198020 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 23265
I20260620 02:58:00.198211 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-1/wal/instance
I20260620 02:58:00.198783 388 external_mini_cluster.cc:1428] Setting key de92bbeeff3d8280a22749b64e34bc95
I20260620 02:58:00.201172 23332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:58:00.210942 23332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36377 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:58:00 dist-test-slave-8p8c krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260620 02:58:00.232728 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.213690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46834 (local address 127.0.97.62:36377)
0620 02:58:00.213873 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:00.213878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:00.215224 (+ 1346us) server_negotiation.cc:407] Connection header received
0620 02:58:00.217183 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:00.217189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:00.217281 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:00.217391 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:00.218948 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.219639 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.221193 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.221611 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.221749 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:00.225380 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:00.225386 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:00.225390 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:00.225427 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:00.228885 (+ 3458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:00.229764 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:00.229773 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:00.229779 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:00.229939 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:00.230522 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:00.230530 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:00.230543 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:00.230960 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:00.231124 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:00.232011 (+ 887us) server_negotiation.cc:299] Negotiation successful
0620 02:58:00.232286 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 02:58:00.233662 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.214237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36377 (local address 127.0.0.1:46834)
0620 02:58:00.215046 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:00.215094 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:00.216896 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:00.217632 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:00.217640 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:00.218159 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:58:00.218778 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:00.218796 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.219806 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.219810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:00.221013 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:00.221030 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.221854 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.221860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:00.222099 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:00.222838 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:58:00.222863 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:58:00.225117 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:58:00.229075 (+ 3958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:00.229083 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:00.229101 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:00.229576 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:00.230095 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:00.230098 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:00.230101 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:00.230313 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:00.231141 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:58:00.231149 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:58:00.231549 (+ 400us) client_negotiation.cc:769] Sending connection context
0620 02:58:00.231842 (+ 293us) client_negotiation.cc:240] Negotiation successful
0620 02:58:00.232300 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":329,"threads_started":1}
I20260620 02:58:00.235704 23332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33505 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:58:00 dist-test-slave-8p8c krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 02:58:00.251727 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.236658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33505 (local address 127.0.0.1:59188)
0620 02:58:00.236889 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:00.236991 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:00.237112 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:00.238483 (+ 1371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:00.238487 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:00.239011 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:58:00.239138 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:00.239146 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.240608 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.240615 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:00.241449 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:00.241460 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.242336 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.242342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:00.242576 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:00.243196 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:58:00.243213 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:58:00.245251 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:58:00.249084 (+ 3833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:00.249090 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:00.249095 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:00.249452 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:00.249917 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:00.249921 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:00.249927 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:00.250047 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:00.250691 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:58:00.250698 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:58:00.250862 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 02:58:00.251000 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 02:58:00.251321 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 02:58:00.251832 23337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.237186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59188 (local address 127.0.97.61:33505)
0620 02:58:00.238014 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:00.238021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:00.238047 (+ 26us) server_negotiation.cc:407] Connection header received
0620 02:58:00.238091 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:00.238096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:00.238284 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:00.238498 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:00.239318 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.240464 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.241587 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.242192 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.242329 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:00.245461 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:00.245472 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:00.245485 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:00.245513 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:00.248914 (+ 3401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:00.249649 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:00.249657 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:00.249661 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:00.249789 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:00.250148 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:00.250156 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:00.250160 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:00.250568 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:00.250788 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:00.251154 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 02:58:00.251440 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":613,"thread_start_us":421,"threads_started":1}
I20260620 02:58:00.254320 23332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43131 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:58:00.255745 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.255168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43131 (local address 127.0.0.1:56166)
0620 02:58:00.255397 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:00.255560 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 02:58:00.255986 23281 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43131: connect: Connection refused (error 111)
I20260620 02:58:00.257711 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.257344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43131 (local address 127.0.0.1:56170)
0620 02:58:00.257514 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:00.257613 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 02:58:00.257944 23332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43131: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43131: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260620 02:58:00.262209 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgxGzum/build/asan/bin/kudu
/tmp/dist-test-taskgxGzum/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:43131
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36377,127.0.97.61:33505,127.0.97.60:43131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 02:58:00.307364 23332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43131 } attempt: 1
I20260620 02:58:00.308660 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.308117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43131 (local address 127.0.0.1:56178)
0620 02:58:00.308421 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:00.308509 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260620 02:58:00.309756 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.309421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43131 (local address 127.0.0.1:56190)
0620 02:58:00.309565 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:00.309650 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 02:58:00.310011 23332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43131: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43131: connect: Connection refused (error 111)
I20260620 02:58:00.378296 23332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43131 } attempt: 2
I20260620 02:58:00.379554 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.379092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43131 (local address 127.0.0.1:56194)
0620 02:58:00.379291 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:00.379381 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 02:58:00.380659 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.380315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43131 (local address 127.0.0.1:56198)
0620 02:58:00.380480 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:00.380551 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 02:58:00.380903 23332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43131: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43131: connect: Connection refused (error 111)
I20260620 02:58:00.388692 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33505 } attempt: 5
Jun 20 02:58:00 dist-test-slave-8p8c krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 02:58:00.401927 23337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.389467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59192 (local address 127.0.97.61:33505)
0620 02:58:00.389630 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:00.389638 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:00.389684 (+ 46us) server_negotiation.cc:407] Connection header received
0620 02:58:00.389912 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:00.389916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:00.389979 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:00.390126 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:00.390963 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.391691 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.392899 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.393345 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.393476 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:00.396684 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:00.396689 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:00.396693 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:00.396724 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:00.399380 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:00.400096 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:00.400100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:00.400102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:00.400216 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:00.400589 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:00.400596 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:00.400600 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:00.400872 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:00.401033 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:00.401387 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 02:58:00.401601 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 02:58:00.401926 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.389366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33505 (local address 127.0.0.1:59192)
0620 02:58:00.389592 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:00.389614 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:00.389761 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:00.390155 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:00.390160 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:00.390635 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:58:00.390812 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:00.390822 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.391832 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.391838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:00.392751 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:00.392762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.393503 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.393517 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:00.393684 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:00.394350 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:58:00.394438 (+ 88us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:58:00.396493 (+ 2055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:58:00.399537 (+ 3044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:00.399542 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:00.399550 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:00.399946 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:00.400325 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:00.400330 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:00.400334 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:00.400473 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:00.400987 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:58:00.400992 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:58:00.401141 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 02:58:00.401282 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 02:58:00.401568 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 02:58:00.403354 23258 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43131 } has no permanent_uuid. Determining permanent_uuid...
I20260620 02:58:00.404318 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.403936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43131 (local address 127.0.0.1:56214)
0620 02:58:00.404106 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:00.404176 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 02:58:00.405197 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.404875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43131 (local address 127.0.0.1:56228)
0620 02:58:00.405026 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:00.405101 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 02:58:00.405414 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43131: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43131: connect: Connection refused (error 111)
I20260620 02:58:00.456645 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43131 } attempt: 1
I20260620 02:58:00.458185 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.457392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43131 (local address 127.0.0.1:56244)
0620 02:58:00.457712 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:00.457861 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260620 02:58:00.459617 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.459105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43131 (local address 127.0.0.1:56250)
0620 02:58:00.459309 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:00.459433 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 02:58:00.459918 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43131: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43131: connect: Connection refused (error 111)
I20260620 02:58:00.472162 23332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43131 } attempt: 3
I20260620 02:58:00.473287 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.472796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43131 (local address 127.0.0.1:56262)
0620 02:58:00.473043 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:00.473133 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 02:58:00.474414 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.473937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43131 (local address 127.0.0.1:56278)
0620 02:58:00.474168 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:00.474248 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260620 02:58:00.474610 23332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43131: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43131: connect: Connection refused (error 111)
I20260620 02:58:00.528205 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43131 } attempt: 2
I20260620 02:58:00.529466 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.528865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43131 (local address 127.0.0.1:56284)
0620 02:58:00.529131 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:00.529253 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 02:58:00.530618 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.530160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43131 (local address 127.0.0.1:56296)
0620 02:58:00.530328 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:00.530453 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 02:58:00.530891 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43131: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43131: connect: Connection refused (error 111)
W20260620 02:58:00.599807 23339 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 02:58:00.600271 23339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 02:58:00.600373 23339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 02:58:00.600456 23339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 02:58:00.607882 23339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 02:58:00.607964 23339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 02:58:00.608021 23339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 02:58:00.608062 23339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 02:58:00.608124 23339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 02:58:00.617821 23332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43131 } attempt: 4
I20260620 02:58:00.617856 23339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36377,127.0.97.61:33505,127.0.97.60:43131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:43131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 02:00:34 UTC on ccycloud.quasar-ycjack.root.comops.site
build id 4456918
ASAN enabled
I20260620 02:58:00.619062 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.618524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43131 (local address 127.0.0.1:56306)
0620 02:58:00.618770 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:00.618875 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 02:58:00.619514 23339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 02:58:00.620164 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.619772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43131 (local address 127.0.0.1:56318)
0620 02:58:00.619942 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:00.620027 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 02:58:00.620402 23332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43131: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43131: connect: Connection refused (error 111)
I20260620 02:58:00.621102 23339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 02:58:00.632717 23347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:00.632802 23345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 02:58:00.632838 23344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 02:58:00.633417 23339 server_base.cc:1034] running on GCE node
I20260620 02:58:00.634145 23339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 02:58:00.635649 23339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 02:58:00.636878 23339 hybrid_clock.cc:648] HybridClock initialized: now 1781924280636815 us; error 27 us; skew 500 ppm
I20260620 02:58:00.637131 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43131 } attempt: 3
I20260620 02:58:00.637285 23339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 02:58:00.638224 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.637712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43131 (local address 127.0.0.1:56320)
0620 02:58:00.637954 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:00.638074 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 02:58:00.639176 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.638824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43131 (local address 127.0.0.1:56328)
0620 02:58:00.638996 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:00.639077 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 02:58:00.639418 23258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43131: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43131: connect: Connection refused (error 111)
Jun 20 02:58:00 dist-test-slave-8p8c krb5kdc[23178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 02:58:00.641314 23339 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260620 02:58:00.642974 23339 webserver.cc:460] Webserver started at http://127.0.97.60:44181/ using document root <none> and password file <none>
I20260620 02:58:00.643639 23339 fs_manager.cc:359] Metadata directory not provided
I20260620 02:58:00.643729 23339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 02:58:00.643975 23339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 02:58:00.646881 23339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-2/data/instance:
uuid: "0924a4178ee14dc0a63d74052ddc8d4b"
format_stamp: "Formatted at 2026-06-20 02:58:00 on dist-test-slave-8p8c"
server_key: "eb8bc9e4417bc4de51ce3737654124d1"
server_key_iv: "d375914cb64c59e64bf1827fb8f93087"
server_key_version: "encryptionkey@0"
I20260620 02:58:00.647603 23339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-2/wal/instance:
uuid: "0924a4178ee14dc0a63d74052ddc8d4b"
format_stamp: "Formatted at 2026-06-20 02:58:00 on dist-test-slave-8p8c"
server_key: "eb8bc9e4417bc4de51ce3737654124d1"
server_key_iv: "d375914cb64c59e64bf1827fb8f93087"
server_key_version: "encryptionkey@0"
I20260620 02:58:00.652683 23339 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 02:58:00.656198 23354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 02:58:00.657570 23339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 02:58:00.657753 23339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-2/wal
uuid: "0924a4178ee14dc0a63d74052ddc8d4b"
format_stamp: "Formatted at 2026-06-20 02:58:00 on dist-test-slave-8p8c"
server_key: "eb8bc9e4417bc4de51ce3737654124d1"
server_key_iv: "d375914cb64c59e64bf1827fb8f93087"
server_key_version: "encryptionkey@0"
I20260620 02:58:00.657965 23339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 02:58:00.689992 23339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 02:58:00.694693 23339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 02:58:00.694990 23339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 02:58:00.713042 23339 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:43131
I20260620 02:58:00.713049 23405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:43131 every 8 connection(s)
I20260620 02:58:00.714458 23339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-2/data/info.pb
I20260620 02:58:00.718048 23406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 02:58:00.719147 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgxGzum/build/asan/bin/kudu as pid 23339
I20260620 02:58:00.719342 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgxGzum/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781923914418765-388-0/minicluster-data/master-2/wal/instance
I20260620 02:58:00.719906 388 external_mini_cluster.cc:1428] Setting key c1a1e3ce6b51eef47be41d1d4f6b0efb
I20260620 02:58:00.727164 23406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36377 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:58:00 dist-test-slave-8p8c krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 02:58:00 dist-test-slave-8p8c krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:58:00.743672 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.721680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56336 (local address 127.0.97.60:43131)
0620 02:58:00.722513 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:00.722525 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:00.722565 (+ 40us) server_negotiation.cc:407] Connection header received
0620 02:58:00.724195 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:00.724216 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:00.724598 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:00.724920 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:00.726653 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.728008 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.729621 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.730489 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.730691 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:00.735266 (+ 4575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:00.735273 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:00.735283 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:00.735319 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:00.739054 (+ 3735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:00.739915 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:00.739921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:00.739928 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:00.740066 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:00.740497 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:00.740514 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:00.740519 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:00.741134 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:00.741368 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:00.741914 (+ 546us) server_negotiation.cc:299] Negotiation successful
0620 02:58:00.742315 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":361,"threads_started":1}
I20260620 02:58:00.746230 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.729851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46848 (local address 127.0.97.62:36377)
0620 02:58:00.730077 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:00.730083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:00.730795 (+ 712us) server_negotiation.cc:407] Connection header received
0620 02:58:00.731079 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:00.731087 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:00.731183 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:00.731262 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:00.732351 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.733473 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.735089 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.735530 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.735631 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:00.739637 (+ 4006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:00.739644 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:00.739651 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:00.739696 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:00.742920 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:00.743871 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:00.743881 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:00.743885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:00.744078 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:00.744555 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:00.744560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:00.744563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:00.744929 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:00.745107 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:00.745701 (+ 594us) server_negotiation.cc:299] Negotiation successful
0620 02:58:00.745888 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 02:58:00.746352 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.729820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36377 (local address 127.0.0.1:46848)
0620 02:58:00.730650 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:00.730707 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:00.730918 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:00.731354 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:00.731362 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:00.731954 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:58:00.732196 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:00.732204 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.733643 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.733648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:00.734945 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:00.734957 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.735741 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.735750 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:00.735940 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:00.736549 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:58:00.736582 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:58:00.739423 (+ 2841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:58:00.743059 (+ 3636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:00.743070 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:00.743077 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:00.743683 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:00.744244 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:00.744247 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:00.744250 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:00.744410 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:00.745140 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:58:00.745148 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:58:00.745370 (+ 222us) client_negotiation.cc:769] Sending connection context
0620 02:58:00.745582 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 02:58:00.745898 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":425,"threads_started":1}
I20260620 02:58:00.747979 23406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33505 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:58:00 dist-test-slave-8p8c krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 20 02:58:00 dist-test-slave-8p8c krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 02:58:00.761152 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.748741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33505 (local address 127.0.0.1:59200)
0620 02:58:00.748916 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:00.748946 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:00.749049 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:00.749339 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:00.749343 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:00.749759 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:58:00.749891 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:00.749899 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.750972 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.750978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:00.751832 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:00.751842 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.752565 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.752571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:00.752760 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:00.753304 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:58:00.753321 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:58:00.755554 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:58:00.758786 (+ 3232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:00.758792 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:00.758796 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:00.759154 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:00.759529 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:00.759536 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:00.759542 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:00.759699 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:00.760202 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:58:00.760206 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:58:00.760352 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 02:58:00.760492 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 02:58:00.760770 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 02:58:00.761196 23337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.748855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59200 (local address 127.0.97.61:33505)
0620 02:58:00.749040 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:00.749046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:00.749065 (+ 19us) server_negotiation.cc:407] Connection header received
0620 02:58:00.749139 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:00.749144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:00.749215 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:00.749318 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:00.750084 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.750818 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.751995 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.752407 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.752550 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:00.755740 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:00.755745 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:00.755748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:00.755780 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:00.758656 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:00.759277 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:00.759289 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:00.759291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:00.759422 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:00.759804 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:00.759811 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:00.759826 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:00.760087 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:00.760238 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:00.760588 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 02:58:00.760790 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 02:58:00.761780 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.747622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46858 (local address 127.0.97.62:36377)
0620 02:58:00.747758 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:00.747762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:00.747775 (+ 13us) server_negotiation.cc:407] Connection header received
0620 02:58:00.747810 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:00.747813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:00.747863 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:00.747970 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:00.749398 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.750120 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.751360 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.751763 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.751903 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:00.756246 (+ 4343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:00.756250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:00.756254 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:00.756283 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:00.758948 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:00.759602 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:00.759608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:00.759618 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:00.759736 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:00.760146 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:00.760152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:00.760156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:00.760416 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:00.760690 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:00.761039 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 02:58:00.761565 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 02:58:00.762914 23406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43131 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 02:58:00 dist-test-slave-8p8c krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 02:58:00 dist-test-slave-8p8c krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 02:58:00.776705 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.763938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56342 (local address 127.0.97.60:43131)
0620 02:58:00.764091 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:00.764096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:00.764113 (+ 17us) server_negotiation.cc:407] Connection header received
0620 02:58:00.764161 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:00.764167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:00.764235 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:00.764323 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:00.765122 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.766108 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.767539 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.768200 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.768337 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:00.771594 (+ 3257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:00.771599 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:00.771602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:00.771629 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:00.774269 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:00.774871 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:00.774878 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:00.774882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:00.774991 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:00.775334 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:00.775338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:00.775341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:00.775606 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:00.775750 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:00.776074 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 02:58:00.776311 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":55}
I20260620 02:58:00.776914 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.763547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43131 (local address 127.0.0.1:56342)
0620 02:58:00.763725 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:00.763744 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:00.763847 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:00.764360 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:00.764365 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:00.764856 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:58:00.764982 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:00.764990 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.766226 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.766232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:00.767402 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:00.767413 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.768384 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.768390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:00.768617 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:00.769225 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:58:00.769247 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:58:00.771423 (+ 2176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:58:00.774408 (+ 2985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:00.774413 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:00.774417 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:00.774762 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:00.775099 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:00.775104 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:00.775107 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:00.775229 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:00.775724 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:58:00.775728 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:58:00.775861 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 02:58:00.775990 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 02:58:00.776443 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":160}
I20260620 02:58:00.778769 23337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.763580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59204 (local address 127.0.97.61:33505)
0620 02:58:00.763742 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:00.763748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:00.764202 (+ 454us) server_negotiation.cc:407] Connection header received
0620 02:58:00.764347 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:00.764352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:00.764425 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:00.764528 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:00.765891 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.766903 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.768389 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.768977 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.769146 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:00.773308 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:00.773312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:00.773316 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:00.773345 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:00.776136 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:00.776771 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:00.776778 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:00.776781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:00.776931 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:00.777376 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:00.777382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:00.777386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:00.777645 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:00.777915 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:00.778210 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 02:58:00.778460 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 02:58:00.782243 23406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0924a4178ee14dc0a63d74052ddc8d4b: Bootstrap starting.
I20260620 02:58:00.783588 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 02:58:00.785742 23406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0924a4178ee14dc0a63d74052ddc8d4b: Neither blocks nor log segments found. Creating new log.
I20260620 02:58:00.786849 23406 log.cc:824] T 00000000000000000000000000000000 P 0924a4178ee14dc0a63d74052ddc8d4b: Log is configured to *not* fsync() on all Append() calls
I20260620 02:58:00.785791 23233 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "72350ceb10504a6aace53b7216b41263"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46858
I20260620 02:58:00.790012 23406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0924a4178ee14dc0a63d74052ddc8d4b: No bootstrap required, opened a new log
I20260620 02:58:00.796921 23406 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0924a4178ee14dc0a63d74052ddc8d4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72350ceb10504a6aace53b7216b41263" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36377 } } peers { permanent_uuid: "85cec0bb4a6844e0bd211af6fd0e2e61" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33505 } } peers { permanent_uuid: "0924a4178ee14dc0a63d74052ddc8d4b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43131 } }
I20260620 02:58:00.797268 23406 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0924a4178ee14dc0a63d74052ddc8d4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0924a4178ee14dc0a63d74052ddc8d4b, State: Initialized, Role: FOLLOWER
I20260620 02:58:00.797704 23406 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0924a4178ee14dc0a63d74052ddc8d4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72350ceb10504a6aace53b7216b41263" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36377 } } peers { permanent_uuid: "85cec0bb4a6844e0bd211af6fd0e2e61" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33505 } } peers { permanent_uuid: "0924a4178ee14dc0a63d74052ddc8d4b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43131 } }
I20260620 02:58:00.798586 23412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0924a4178ee14dc0a63d74052ddc8d4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72350ceb10504a6aace53b7216b41263" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36377 } } peers { permanent_uuid: "85cec0bb4a6844e0bd211af6fd0e2e61" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33505 } } peers { permanent_uuid: "0924a4178ee14dc0a63d74052ddc8d4b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43131 } } }
I20260620 02:58:00.798887 23412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0924a4178ee14dc0a63d74052ddc8d4b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:58:00.799810 23406 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0924a4178ee14dc0a63d74052ddc8d4b [sys.catalog]: configured and running, proceeding with master startup.
W20260620 02:58:00.809185 23423 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0924a4178ee14dc0a63d74052ddc8d4b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 02:58:00.809350 23423 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:58:00.806953 23339 master_runner.cc:186] Error getting master registration for 127.0.97.62:36377: OK, instance_id { permanent_uuid: "72350ceb10504a6aace53b7216b41263" instance_seqno: 1781924279666162 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 02:58:00.816205 23339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 02:58:00.816649 23258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43131 } attempt: 4
Jun 20 02:58:00 dist-test-slave-8p8c krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 02:58:00.830801 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.817249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43131 (local address 127.0.0.1:56358)
0620 02:58:00.817433 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:00.817453 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:00.817570 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:00.817884 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:00.817889 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:00.818318 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:58:00.818545 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:00.818555 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.819568 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.819574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:00.820501 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:00.820512 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.821313 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.821319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:00.821527 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:00.822148 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:58:00.822166 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:58:00.824648 (+ 2482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:58:00.827834 (+ 3186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:00.827840 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:00.827844 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:00.828347 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:00.828806 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:00.828812 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:00.828816 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:00.828998 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:00.829709 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:58:00.829714 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:58:00.829876 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 02:58:00.830048 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 02:58:00.830386 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 02:58:00.831015 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.817325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56358 (local address 127.0.97.60:43131)
0620 02:58:00.817546 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:00.817552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:00.817580 (+ 28us) server_negotiation.cc:407] Connection header received
0620 02:58:00.817677 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:00.817685 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:00.817755 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:00.817856 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:00.818682 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.819420 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.820650 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.821149 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.821298 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:00.824814 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:00.824819 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:00.824823 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:00.824850 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:00.827687 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:00.828489 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:00.828499 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:00.828504 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:00.828674 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:00.829143 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:00.829148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:00.829152 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:00.829575 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:00.829740 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:00.830258 (+ 518us) server_negotiation.cc:299] Negotiation successful
0620 02:58:00.830525 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 02:58:00.835815 23258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72350ceb10504a6aace53b7216b41263: Bootstrap starting.
I20260620 02:58:00.839533 23258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72350ceb10504a6aace53b7216b41263: Neither blocks nor log segments found. Creating new log.
I20260620 02:58:00.840619 23258 log.cc:824] T 00000000000000000000000000000000 P 72350ceb10504a6aace53b7216b41263: Log is configured to *not* fsync() on all Append() calls
I20260620 02:58:00.843676 23258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72350ceb10504a6aace53b7216b41263: No bootstrap required, opened a new log
I20260620 02:58:00.849315 23258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72350ceb10504a6aace53b7216b41263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72350ceb10504a6aace53b7216b41263" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36377 } } peers { permanent_uuid: "85cec0bb4a6844e0bd211af6fd0e2e61" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33505 } } peers { permanent_uuid: "0924a4178ee14dc0a63d74052ddc8d4b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43131 } }
I20260620 02:58:00.849705 23258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72350ceb10504a6aace53b7216b41263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72350ceb10504a6aace53b7216b41263, State: Initialized, Role: FOLLOWER
I20260620 02:58:00.850281 23258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72350ceb10504a6aace53b7216b41263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72350ceb10504a6aace53b7216b41263" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36377 } } peers { permanent_uuid: "85cec0bb4a6844e0bd211af6fd0e2e61" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33505 } } peers { permanent_uuid: "0924a4178ee14dc0a63d74052ddc8d4b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43131 } }
I20260620 02:58:00.851372 23426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72350ceb10504a6aace53b7216b41263 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72350ceb10504a6aace53b7216b41263" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36377 } } peers { permanent_uuid: "85cec0bb4a6844e0bd211af6fd0e2e61" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33505 } } peers { permanent_uuid: "0924a4178ee14dc0a63d74052ddc8d4b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43131 } } }
I20260620 02:58:00.851729 23426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72350ceb10504a6aace53b7216b41263 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:58:00.852530 23258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72350ceb10504a6aace53b7216b41263 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:58:00.852672 23233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72350ceb10504a6aace53b7216b41263 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 02:58:00.852901 23233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72350ceb10504a6aace53b7216b41263 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:58:00.856660 23233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72350ceb10504a6aace53b7216b41263 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72350ceb10504a6aace53b7216b41263" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36377 } } peers { permanent_uuid: "85cec0bb4a6844e0bd211af6fd0e2e61" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33505 } } peers { permanent_uuid: "0924a4178ee14dc0a63d74052ddc8d4b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43131 } }
I20260620 02:58:00.858318 23233 leader_election.cc:288] T 00000000000000000000000000000000 P 72350ceb10504a6aace53b7216b41263 [CANDIDATE]: Term 1 election: Requested vote from peers 85cec0bb4a6844e0bd211af6fd0e2e61 (127.0.97.61:33505), 0924a4178ee14dc0a63d74052ddc8d4b (127.0.97.60:43131)
I20260620 02:58:00.861680 23381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "72350ceb10504a6aace53b7216b41263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0924a4178ee14dc0a63d74052ddc8d4b"
I20260620 02:58:00.862211 23381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0924a4178ee14dc0a63d74052ddc8d4b [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:58:00.865700 23381 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0924a4178ee14dc0a63d74052ddc8d4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72350ceb10504a6aace53b7216b41263 in term 1.
I20260620 02:58:00.866705 23207 leader_election.cc:302] T 00000000000000000000000000000000 P 72350ceb10504a6aace53b7216b41263 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0924a4178ee14dc0a63d74052ddc8d4b, 72350ceb10504a6aace53b7216b41263; no voters:
I20260620 02:58:00.867195 23426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72350ceb10504a6aace53b7216b41263 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 02:58:00.867967 23426 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72350ceb10504a6aace53b7216b41263 [term 1 LEADER]: Becoming Leader. State: Replica: 72350ceb10504a6aace53b7216b41263, State: Running, Role: LEADER
I20260620 02:58:00.868697 23426 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72350ceb10504a6aace53b7216b41263 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72350ceb10504a6aace53b7216b41263" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36377 } } peers { permanent_uuid: "85cec0bb4a6844e0bd211af6fd0e2e61" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33505 } } peers { permanent_uuid: "0924a4178ee14dc0a63d74052ddc8d4b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43131 } }
I20260620 02:58:00.873169 23437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72350ceb10504a6aace53b7216b41263 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72350ceb10504a6aace53b7216b41263. Latest consensus state: current_term: 1 leader_uuid: "72350ceb10504a6aace53b7216b41263" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72350ceb10504a6aace53b7216b41263" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36377 } } peers { permanent_uuid: "85cec0bb4a6844e0bd211af6fd0e2e61" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33505 } } peers { permanent_uuid: "0924a4178ee14dc0a63d74052ddc8d4b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43131 } } }
I20260620 02:58:00.873669 23437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72350ceb10504a6aace53b7216b41263 [sys.catalog]: This master's current role is: LEADER
W20260620 02:58:00.903825 23442 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 72350ceb10504a6aace53b7216b41263: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 02:58:00.885833 23191 master_runner.cc:186] Error getting master registration for 127.0.97.61:33505: OK, instance_id { permanent_uuid: "85cec0bb4a6844e0bd211af6fd0e2e61" instance_seqno: 1781924280189556 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260620 02:58:00.904516 23212 tablet.cc:2369] T 00000000000000000000000000000000 P 72350ceb10504a6aace53b7216b41263: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 02:58:00.905479 23191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 02:58:00.906481 23442 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 02:58:00.906934 23440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 02:58:00.909263 23440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 02:58:00.919592 23332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43131 } attempt: 5
I20260620 02:58:00.927824 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.898673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46866 (local address 127.0.97.62:36377)
0620 02:58:00.898851 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:00.898858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:00.908842 (+ 9984us) server_negotiation.cc:407] Connection header received
0620 02:58:00.909017 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:00.909022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:00.909120 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:00.909221 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:00.911392 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.912630 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.914774 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.915528 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.915650 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:00.920169 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:00.920179 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:00.920187 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:00.920244 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:00.924141 (+ 3897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:00.925072 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:00.925078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:00.925084 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:00.925287 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:00.925829 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:00.925835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:00.925841 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:00.926243 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:00.926539 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:00.927174 (+ 635us) server_negotiation.cc:299] Negotiation successful
0620 02:58:00.927431 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 02:58:00.938058 23337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.899130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59210 (local address 127.0.97.61:33505)
0620 02:58:00.899308 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:00.899313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:00.908167 (+ 8854us) server_negotiation.cc:407] Connection header received
0620 02:58:00.908379 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:00.908387 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:00.908492 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:00.908602 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:00.920320 (+ 11718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.921543 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.923655 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.924356 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.924454 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:00.930853 (+ 6399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:00.930860 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:00.930866 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:00.930910 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:00.934611 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:00.935477 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:00.935483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:00.935487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:00.935647 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:00.936091 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:00.936097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:00.936100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:00.936526 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:00.936773 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:00.937431 (+ 658us) server_negotiation.cc:299] Negotiation successful
0620 02:58:00.937685 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 02:58:00.947191 23360 tablet.cc:2369] T 00000000000000000000000000000000 P 0924a4178ee14dc0a63d74052ddc8d4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 02:58:00 dist-test-slave-8p8c krb5kdc[23178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781924280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 02:58:00.958645 23381 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0924a4178ee14dc0a63d74052ddc8d4b [term 1 FOLLOWER]: Refusing update from remote peer 72350ceb10504a6aace53b7216b41263: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:58:00.959872 23437 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 72350ceb10504a6aace53b7216b41263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0924a4178ee14dc0a63d74052ddc8d4b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 02:58:00.961314 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.920277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43131 (local address 127.0.0.1:56380)
0620 02:58:00.920761 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 02:58:00.920785 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 02:58:00.920913 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 02:58:00.943407 (+ 22494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 02:58:00.943412 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 02:58:00.943930 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0620 02:58:00.944136 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:00.944150 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.946938 (+ 2788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.946944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:00.948378 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 02:58:00.948394 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.949493 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.949499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 02:58:00.949729 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:00.950421 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 02:58:00.950452 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 02:58:00.953496 (+ 3044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 02:58:00.958102 (+ 4606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:00.958108 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:00.958112 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:00.958673 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:00.959281 (+ 608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 02:58:00.959286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 02:58:00.959293 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 02:58:00.959454 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 02:58:00.960295 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 02:58:00.960301 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 02:58:00.960461 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 02:58:00.960624 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 02:58:00.960919 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292}
I20260620 02:58:00.962801 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.921160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56380 (local address 127.0.97.60:43131)
0620 02:58:00.943001 (+ 21841us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:00.943010 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:00.943042 (+ 32us) server_negotiation.cc:407] Connection header received
0620 02:58:00.943109 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:00.943115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:00.943206 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:00.944299 (+ 1093us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:00.944630 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.946778 (+ 2148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.948661 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.949304 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.950038 (+ 734us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:00.953760 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:00.953767 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:00.953773 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:00.953866 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:00.957921 (+ 4055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:00.958959 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:00.958964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:00.958968 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:00.959126 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:00.959613 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:00.959618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:00.959622 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:00.960043 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:00.961962 (+ 1919us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:00.962100 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 02:58:00.962318 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":21673,"thread_start_us":21442,"threads_started":1}
I20260620 02:58:00.969699 23332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85cec0bb4a6844e0bd211af6fd0e2e61: Bootstrap starting.
I20260620 02:58:00.975024 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:00.908551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56374 (local address 127.0.97.60:43131)
0620 02:58:00.908739 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:00.908747 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:00.918251 (+ 9504us) server_negotiation.cc:407] Connection header received
0620 02:58:00.918698 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:00.918706 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:00.918791 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:00.918864 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:00.929603 (+ 10739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.930776 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.944497 (+ 13721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:00.945404 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:00.945534 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:00.966098 (+ 20564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 02:58:00.966104 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 02:58:00.966108 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 02:58:00.966140 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 02:58:00.970195 (+ 4055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:00.971703 (+ 1508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:00.971709 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:00.971714 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:00.971851 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 02:58:00.972561 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 02:58:00.972568 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 02:58:00.972572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 02:58:00.972937 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 02:58:00.973185 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:00.974069 (+ 884us) server_negotiation.cc:299] Negotiation successful
0620 02:58:00.974313 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":65}
I20260620 02:58:00.977095 23332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85cec0bb4a6844e0bd211af6fd0e2e61: Neither blocks nor log segments found. Creating new log.
I20260620 02:58:00.978806 23332 log.cc:824] T 00000000000000000000000000000000 P 85cec0bb4a6844e0bd211af6fd0e2e61: Log is configured to *not* fsync() on all Append() calls
I20260620 02:58:00.988801 23332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85cec0bb4a6844e0bd211af6fd0e2e61: No bootstrap required, opened a new log
I20260620 02:58:01.001524 23412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0924a4178ee14dc0a63d74052ddc8d4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72350ceb10504a6aace53b7216b41263. Latest consensus state: current_term: 1 leader_uuid: "72350ceb10504a6aace53b7216b41263" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72350ceb10504a6aace53b7216b41263" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36377 } } peers { permanent_uuid: "85cec0bb4a6844e0bd211af6fd0e2e61" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33505 } } peers { permanent_uuid: "0924a4178ee14dc0a63d74052ddc8d4b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43131 } } }
I20260620 02:58:01.001847 23412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0924a4178ee14dc0a63d74052ddc8d4b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:58:01.001045 23332 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85cec0bb4a6844e0bd211af6fd0e2e61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72350ceb10504a6aace53b7216b41263" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36377 } } peers { permanent_uuid: "85cec0bb4a6844e0bd211af6fd0e2e61" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33505 } } peers { permanent_uuid: "0924a4178ee14dc0a63d74052ddc8d4b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43131 } }
I20260620 02:58:01.002064 23332 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85cec0bb4a6844e0bd211af6fd0e2e61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85cec0bb4a6844e0bd211af6fd0e2e61, State: Initialized, Role: FOLLOWER
I20260620 02:58:01.005658 23332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85cec0bb4a6844e0bd211af6fd0e2e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72350ceb10504a6aace53b7216b41263" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36377 } } peers { permanent_uuid: "85cec0bb4a6844e0bd211af6fd0e2e61" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33505 } } peers { permanent_uuid: "0924a4178ee14dc0a63d74052ddc8d4b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43131 } }
I20260620 02:58:01.009289 23449 mvcc.cc:205] Tried to move back new op lower bound from 7298761854782902272 to 7298761854444855296. Current Snapshot: MvccSnapshot[applied={T|T < 7298761854782902272}]
I20260620 02:58:01.012408 23451 mvcc.cc:205] Tried to move back new op lower bound from 7298761854782902272 to 7298761854444855296. Current Snapshot: MvccSnapshot[applied={T|T < 7298761854782902272 or (T in {7298761854782902272})}]
I20260620 02:58:01.013059 23412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0924a4178ee14dc0a63d74052ddc8d4b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "72350ceb10504a6aace53b7216b41263" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72350ceb10504a6aace53b7216b41263" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36377 } } peers { permanent_uuid: "85cec0bb4a6844e0bd211af6fd0e2e61" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33505 } } peers { permanent_uuid: "0924a4178ee14dc0a63d74052ddc8d4b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43131 } } }
I20260620 02:58:01.013327 23412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0924a4178ee14dc0a63d74052ddc8d4b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:58:01.016615 23454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85cec0bb4a6844e0bd211af6fd0e2e61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72350ceb10504a6aace53b7216b41263" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36377 } } peers { permanent_uuid: "85cec0bb4a6844e0bd211af6fd0e2e61" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33505 } } peers { permanent_uuid: "0924a4178ee14dc0a63d74052ddc8d4b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43131 } } }
I20260620 02:58:01.018522 23437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72350ceb10504a6aace53b7216b41263 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "72350ceb10504a6aace53b7216b41263" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72350ceb10504a6aace53b7216b41263" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36377 } } peers { permanent_uuid: "85cec0bb4a6844e0bd211af6fd0e2e61" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33505 } } peers { permanent_uuid: "0924a4178ee14dc0a63d74052ddc8d4b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43131 } } }
I20260620 02:58:01.018795 23437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72350ceb10504a6aace53b7216b41263 [sys.catalog]: This master's current role is: LEADER
I20260620 02:58:01.019091 23454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85cec0bb4a6844e0bd211af6fd0e2e61 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:58:01.018965 23332 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85cec0bb4a6844e0bd211af6fd0e2e61 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 02:58:01.019019 23307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "72350ceb10504a6aace53b7216b41263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "85cec0bb4a6844e0bd211af6fd0e2e61"
I20260620 02:58:01.019371 23306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85cec0bb4a6844e0bd211af6fd0e2e61 [term 0 FOLLOWER]: Advancing to term 1
I20260620 02:58:01.023308 23306 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 85cec0bb4a6844e0bd211af6fd0e2e61 [term 1 FOLLOWER]: Refusing update from remote peer 72350ceb10504a6aace53b7216b41263: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 02:58:01.026814 23307 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 85cec0bb4a6844e0bd211af6fd0e2e61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72350ceb10504a6aace53b7216b41263 in term 1.
I20260620 02:58:01.029788 23440 catalog_manager.cc:1347] Generated new cluster ID: 2fcfbe25075b4463bcfe8002bab1b785
I20260620 02:58:01.029913 23440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 02:58:01.029917 23426 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 72350ceb10504a6aace53b7216b41263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85cec0bb4a6844e0bd211af6fd0e2e61" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260620 02:58:01.066460 23454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85cec0bb4a6844e0bd211af6fd0e2e61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72350ceb10504a6aace53b7216b41263. Latest consensus state: current_term: 1 leader_uuid: "72350ceb10504a6aace53b7216b41263" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72350ceb10504a6aace53b7216b41263" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36377 } } peers { permanent_uuid: "85cec0bb4a6844e0bd211af6fd0e2e61" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33505 } } peers { permanent_uuid: "0924a4178ee14dc0a63d74052ddc8d4b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43131 } } }
I20260620 02:58:01.066845 23454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85cec0bb4a6844e0bd211af6fd0e2e61 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:58:01.068068 23462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85cec0bb4a6844e0bd211af6fd0e2e61 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "72350ceb10504a6aace53b7216b41263" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72350ceb10504a6aace53b7216b41263" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36377 } } peers { permanent_uuid: "85cec0bb4a6844e0bd211af6fd0e2e61" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33505 } } peers { permanent_uuid: "0924a4178ee14dc0a63d74052ddc8d4b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43131 } } }
I20260620 02:58:01.068507 23462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85cec0bb4a6844e0bd211af6fd0e2e61 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 02:58:01.073563 23437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72350ceb10504a6aace53b7216b41263 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "72350ceb10504a6aace53b7216b41263" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72350ceb10504a6aace53b7216b41263" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36377 } } peers { permanent_uuid: "85cec0bb4a6844e0bd211af6fd0e2e61" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33505 } } peers { permanent_uuid: "0924a4178ee14dc0a63d74052ddc8d4b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43131 } } }
I20260620 02:58:01.073868 23437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72350ceb10504a6aace53b7216b41263 [sys.catalog]: This master's current role is: LEADER
I20260620 02:58:01.080797 23440 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 02:58:01.082705 23440 catalog_manager.cc:1504] Loading token signing keys...
I20260620 02:58:01.099802 23440 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72350ceb10504a6aace53b7216b41263: Generated new TSK 0
I20260620 02:58:01.101024 23440 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 02:58:01.105484 23472 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260620 02:58:01.177218 23261 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:58:01.163913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46874 (local address 127.0.97.62:36377)
0620 02:58:01.164185 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:01.164193 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:01.175037 (+ 10844us) server_negotiation.cc:407] Connection header received
0620 02:58:01.175244 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:01.175254 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:01.175393 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:01.175535 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:01.176916 (+ 1381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46874: BlockingRecv error: recv got EOF from 127.0.0.1:46874 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 02:58:01.180176 23337 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:58:01.163688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59224 (local address 127.0.97.61:33505)
0620 02:58:01.163892 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:01.163899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:01.176793 (+ 12894us) server_negotiation.cc:407] Connection header received
0620 02:58:01.178589 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:01.178596 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:01.178692 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:01.178769 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:01.179943 (+ 1174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59224: BlockingRecv error: recv got EOF from 127.0.0.1:59224 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 02:58:01.182030 23409 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:58:01.177369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56384 (local address 127.0.97.60:43131)
0620 02:58:01.177555 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:01.177563 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:01.179725 (+ 2162us) server_negotiation.cc:407] Connection header received
0620 02:58:01.179967 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:01.179973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:01.180054 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:01.180128 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:01.181619 (+ 1491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56384: BlockingRecv error: recv got EOF from 127.0.0.1:56384 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 02:58:01.189689 23337 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:58:01.187795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59232 (local address 127.0.97.61:33505)
0620 02:58:01.187988 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:01.187993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:01.188188 (+ 195us) server_negotiation.cc:407] Connection header received
0620 02:58:01.188442 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:01.188452 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:01.188558 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:01.188676 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:01.189472 (+ 796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59232: BlockingRecv error: recv got EOF from 127.0.0.1:59232 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 02:58:01.200963 23337 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:58:01.194903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59244 (local address 127.0.97.61:33505)
0620 02:58:01.195128 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:01.195137 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:01.199372 (+ 4235us) server_negotiation.cc:407] Connection header received
0620 02:58:01.199605 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:01.199619 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:01.199714 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:01.199830 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:01.200751 (+ 921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59244: BlockingRecv error: recv got EOF from 127.0.0.1:59244 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 02:58:01.230876 23337 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:58:01.229016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59252 (local address 127.0.97.61:33505)
0620 02:58:01.229214 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:01.229221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:01.229245 (+ 24us) server_negotiation.cc:407] Connection header received
0620 02:58:01.229555 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:01.229562 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:01.229664 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:01.229788 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:01.230628 (+ 840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59252: BlockingRecv error: recv got EOF from 127.0.0.1:59252 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 02:58:01.240690 23337 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:58:01.238425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59260 (local address 127.0.97.61:33505)
0620 02:58:01.238645 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:01.238654 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:01.239220 (+ 566us) server_negotiation.cc:407] Connection header received
0620 02:58:01.239437 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:01.239443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:01.239538 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:01.239646 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:01.240467 (+ 821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59260: BlockingRecv error: recv got EOF from 127.0.0.1:59260 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
W20260620 02:58:01.254915 23337 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:58:01.252527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59276 (local address 127.0.97.61:33505)
0620 02:58:01.252749 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:01.252755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:01.253316 (+ 561us) server_negotiation.cc:407] Connection header received
0620 02:58:01.253516 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:01.253522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:01.253638 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:01.253775 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:01.254676 (+ 901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59276: BlockingRecv error: recv got EOF from 127.0.0.1:59276 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260620 02:58:01.278599 23337 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:58:01.276094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59286 (local address 127.0.97.61:33505)
0620 02:58:01.276298 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:01.276304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:01.276893 (+ 589us) server_negotiation.cc:407] Connection header received
0620 02:58:01.277125 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:01.277131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:01.277253 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:01.277382 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:01.278317 (+ 935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59286: BlockingRecv error: recv got EOF from 127.0.0.1:59286 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260620 02:58:01.317471 23337 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:58:01.315051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59300 (local address 127.0.97.61:33505)
0620 02:58:01.315249 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:01.315255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:01.315864 (+ 609us) server_negotiation.cc:407] Connection header received
0620 02:58:01.316069 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:01.316078 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:01.316184 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:01.316303 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:01.317185 (+ 882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59300: BlockingRecv error: recv got EOF from 127.0.0.1:59300 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 02:58:01.387316 23337 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:58:01.385313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59314 (local address 127.0.97.61:33505)
0620 02:58:01.385513 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:01.385519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:01.385880 (+ 361us) server_negotiation.cc:407] Connection header received
0620 02:58:01.386055 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:01.386062 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:01.386166 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:01.386292 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:01.387099 (+ 807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59314: BlockingRecv error: recv got EOF from 127.0.0.1:59314 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260620 02:58:01.520982 23337 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:58:01.518813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59320 (local address 127.0.97.61:33505)
0620 02:58:01.519072 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:01.519081 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:01.519457 (+ 376us) server_negotiation.cc:407] Connection header received
0620 02:58:01.519721 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:01.519725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:01.519809 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:01.519918 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:01.520697 (+ 779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59320: BlockingRecv error: recv got EOF from 127.0.0.1:59320 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260620 02:58:01.782509 23337 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 02:58:01.780483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59336 (local address 127.0.97.61:33505)
0620 02:58:01.780667 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:01.780672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:01.781101 (+ 429us) server_negotiation.cc:407] Connection header received
0620 02:58:01.781286 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:01.781290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:01.781361 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:01.781455 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 02:58:01.782245 (+ 790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59336: BlockingRecv error: recv got EOF from 127.0.0.1:59336 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 02:58:01.811297 23423 catalog_manager.cc:1259] Loaded cluster ID: 2fcfbe25075b4463bcfe8002bab1b785
I20260620 02:58:01.811405 23423 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0924a4178ee14dc0a63d74052ddc8d4b: loading cluster ID for follower catalog manager: success
I20260620 02:58:01.813859 23423 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0924a4178ee14dc0a63d74052ddc8d4b: acquiring CA information for follower catalog manager: success
I20260620 02:58:01.815680 23423 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0924a4178ee14dc0a63d74052ddc8d4b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:58:02.110061 23472 catalog_manager.cc:1259] Loaded cluster ID: 2fcfbe25075b4463bcfe8002bab1b785
I20260620 02:58:02.110173 23472 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 85cec0bb4a6844e0bd211af6fd0e2e61: loading cluster ID for follower catalog manager: success
I20260620 02:58:02.112736 23472 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 85cec0bb4a6844e0bd211af6fd0e2e61: acquiring CA information for follower catalog manager: success
I20260620 02:58:02.114771 23472 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 85cec0bb4a6844e0bd211af6fd0e2e61: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 02:58:02.305488 23539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:02.298276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54478 (local address 127.0.97.61:33505)
0620 02:58:02.299178 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:02.299186 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:02.299213 (+ 27us) server_negotiation.cc:407] Connection header received
0620 02:58:02.299283 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:02.299287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:02.299460 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:02.299617 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:58:02.300167 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:02.301208 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:02.302774 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:02.303361 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:02.303543 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:02.304140 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:58:02.304740 (+ 600us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:58:02.304918 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:02.305011 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 02:58:02.305109 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":572,"threads_started":1}
I20260620 02:58:02.319504 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:02.312103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52990 (local address 127.0.97.60:43131)
0620 02:58:02.312675 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:02.312682 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:02.313061 (+ 379us) server_negotiation.cc:407] Connection header received
0620 02:58:02.313339 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:02.313343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:02.313506 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:02.313651 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:58:02.314121 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:02.314922 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:02.316912 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:02.317332 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:02.317429 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:02.318121 (+ 692us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:58:02.318743 (+ 622us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:58:02.318872 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:02.319030 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 02:58:02.319099 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":298,"threads_started":1}
I20260620 02:58:02.343271 23539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:02.328668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54494 (local address 127.0.97.61:33505)
0620 02:58:02.328884 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:02.328891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:02.335991 (+ 7100us) server_negotiation.cc:407] Connection header received
0620 02:58:02.336307 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:02.336314 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:02.336418 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:02.336489 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:58:02.337368 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:02.338558 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:02.340934 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:02.341616 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:02.341719 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:02.342339 (+ 620us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:58:02.342510 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:58:02.342585 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:02.342837 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 02:58:02.342941 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 02:58:02.343693 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:02.335621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53004 (local address 127.0.97.60:43131)
0620 02:58:02.335830 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:02.335839 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:02.336618 (+ 779us) server_negotiation.cc:407] Connection header received
0620 02:58:02.336928 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:02.336935 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:02.337053 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:02.337117 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:58:02.338022 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:02.339058 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:02.341444 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:02.342077 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:02.342177 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:02.342894 (+ 717us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:58:02.343016 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:58:02.343077 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:02.343292 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 02:58:02.343373 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 02:58:02.344357 23552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 02:58:02.328388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54574 (local address 127.0.97.62:36377)
0620 02:58:02.334793 (+ 6405us) server_negotiation.cc:206] Beginning negotiation
0620 02:58:02.334801 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 02:58:02.336427 (+ 1626us) server_negotiation.cc:407] Connection header received
0620 02:58:02.336561 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 02:58:02.336569 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 02:58:02.336799 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 02:58:02.336968 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 02:58:02.337816 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:02.339005 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:02.341447 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 02:58:02.342077 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 02:58:02.342192 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 02:58:02.342967 (+ 775us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 02:58:02.343411 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 02:58:02.343513 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 02:58:02.343947 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 02:58:02.344045 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6253,"thread_start_us":6145,"threads_started":1}
I20260620 02:58:02.349633 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 23191
I20260620 02:58:02.350831 23253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:58:02.468493 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 23191
I20260620 02:58:02.490263 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 23265
I20260620 02:58:02.491509 23327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:58:02.600260 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 23265
I20260620 02:58:02.620592 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgxGzum/build/asan/bin/kudu pid 23339
I20260620 02:58:02.621722 23401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 02:58:02.730454 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgxGzum/build/asan/bin/kudu with pid 23339
2026-06-20T02:58:02Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4906 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9843 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (368273 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST